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ABSTRACT 


The  use  and  scope  of  LADAR  applications  continues  to  expand  as  the  technology  matures,  but 
standard  protocols  or  procedures  for  calibrating  and  testing  of  LADARs  have  yet  to  be 
developed.  While  selections  of  LADAR  instruments  are  generally  based  on  the  manufacturer’s 
specifications,  there  are  no  uniform  definitions  of  such  specifications  nor  are  there  uniform 
guidelines  for  their  validation.  As  a first  step  towards  developing  agreement  on  these  issues,  a 
LADAR  Calibration  Facility  workshop  was  convened  at  NIST  on  June  12-13,  2003.  The 
proceedings  of  the  workshop  are  presented  in  this  report. 
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DISCLAIMER 

Certain  trade  names  and  company  products  are  mentioned  in  the  text  or  identified  in  an 
illustration.  In  no  case  does  such  an  identification  imply  recommendation  or  endorsement  by  the 
National  Institute  of  Standards  and  Technology,  nor  does  it  imply  that  the  products  are 
necessarily  the  best  available  for  the  purpose. 

The  opinions  expressed  in  the  presentations  and  papers  by  non-NIST  authors  are  those  of  the 
non-NIST  authors  and  are  not  necessarily  the  opinions  of  NIST. 


POLICY 

It  is  NIST’s  policy  to  use  the  International  System  of  Units  (SI).  However,  some  of  the  units 
used  in  the  workshop  presentations  and  papers  are  in  U.S.  customary  units  because  of  the 
intended  audience.  Conversions  from  the  U.  S.  customary  units  to  SI  have  been  made  where 
possible. 
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1. 


INTRODUCTION 


The  Building  and  Fire  Research  Laboratory  (BFRL)  of  the  National  Institute  of  Standards  and 
Technology  (NIST)  conducted  a workshop  as  a first  step  towards  the  establishment  of  a LADAR 
calibration  facility  on  June  12-13,  2003  at  the  NIST  campus  in  Gaithersburg,  MD  (LADAR  is 
laser  distance  and  ranging).  The  objectives  of  the  workshop  were: 

- to  provide  a forum  for  sharing  and  discussing  current  efforts  in  LADAR  calibration 

- to  determine  the  types  of  performance  evaluations  and  test  protocols  required 

- to  identify  the  physical  requirements  of  a calibration  facility 

- to  explore  potential  plans  for  the  establishment  /operation  /location  of  a LADAR  test 
facility 

At  NIST,  the  growing  use  of  LADAR  technology  and  LADAR  data  processing  underscores  the 
necessity  of  an  intramural  test  facility.  In  keeping  with  its  mission  as  the  Nation’s  metrology 
laboratory,  NIST  is  in  a position  to  provide  metrology  support  to  both  users  and  manufacturers  of 
LADARs  in  addition  to  meeting  its  own  substantial  internal  calibration  needs. 

The  workshop  was  organized  around  six  presentations  by  leading  proponents  of  LADAR 
technology.  The  presentations  were  followed  by  three  break-out  group  discussions  on  both  days. 
The  discussions  reinforced  the  realization  that  this  burgeoning  field  of  development  and 
application  requires  metrology  support. 

These  proceedings  are  organized  as  follows:  Chapter  2 - workshop  agenda,  Chapter  3 - 
summaries  of  break-out  group  discussions,  Chapter  4 - workshop  summary  and  future  steps, 
Appendix  A - List  of  participants,  and  Appendices  B to  H - workshop  presentations  or  papers. 


1 


2. 


WORKSHOP  AGENDA 


June 

8:00  - 
8:15  - 

8:30- 

8:40- 


9:00- 


9:30- 

10:00 


12,  2003 

8:15  Registration 

8:30  Introduction 

8:40  Welcome 


9:00  Overview:  NIST/Construction  Metrology  and 
Automation’s  Work 

The  NIST  Materials  and  Construction  Research  Division 
(MCRD)  and  the  Intelligent  Systems  Division  (ISD)  make 
daily  use  of  an  extensive  array  of  LADAR  systems  for  such 
widely  varying  applications  as  real-time  autonomous 
control  of  machines  to  tracking  of  manufactured 
components  and  characterization  of  construction  sites  for 
automated  assessment  of  job  status.  Significant  recurring 
calibration  needs  by  the  disparate  LADAR  user- 
communities  at  NIST  has  led  to  the  concept  of  a joint 
calibration  facility  which,  with  industry  input,  may  also 
serve  as  a neutral  national  calibration  resource. 

9:30  Calibration  Activities  at  the  National  Research 
Council  of  Canada  (NRCC) 

The  NRCC  has  been  involved  in  the  field  of  3D  Imaging 
for  more  than  20  years.  Various  aspects  of  3D  have  been 
investigated,  from  laser  scanner  development,  calibration, 
performance  assessment,  3D  geometrical  processing, 
display,  and  various  industrial  applications.  This 
presentation  will  focus  on  the  activities  related  to  the 
calibration  of  laser  scanners  and  the  creation  of  a 
calibration  facility  and  tools  that  allow  accurate  modeling 
and  evaluation  of  the  performances  of  3D  systems. 

1 0:00  Future  of  Laser  Radar 
-10:10  Break 


G.  Cheok 
NIST 

J.  Snell 
NIST 

B.  Stone 
NIST 


F.  Blais 
NRCC 


G.  Kamerman 
FastMetrix 
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10:10-12:00 

3 Break  out  groups 

12:00-12:30 

Reconvene  for  whole  group  discussion 

12:30-1:30 

Lunch  - (Self-service  NIST  cafeteria) 

1:30-2:00 

LADAR  Measurements  at  Boeing  J.  Palmateer 

Boeing 

Current  Boeing  efforts  in  LADAR  development  have  been 
directed  toward  systems  capable  of  measuring  parts  and 
assemblies  in  a manufacturing  environment.  Goals 

include  on-machine  measurement  and  measurement 
during  assembly  for  control  and  inspection.  A national 
laboratory  with  expertise  in  various  LADAR  systems, 
which  will  ensure  a better  understanding  of  the  modes  of 

LADAR  failure  is  envisioned. 

2:00-2:30 

Corps  of  Engineers  Field  Measurement  Requirements  C.  Daniel 

USACE 

The  rapid  assessment,  development,  and  insertion  of 

LADAR  calibration  standards  into  the  Corps'  Operations 
and  Maintenance  programs  will  improve  the  operational 
effectiveness  of  a wide  range  of  traditional  field 
measurement  requirements.  The  presentation  will 
describe  the  results  of  recent  Civil  Works  projects  where 
commercially  available  laser  scanning  systems  have  been 
used.  A description,  with  illustrative  examples  of  products 
from  the  projects,  initial  accuracy  results,  and  time 
response-related  results  with  future  implications,  are 
presented. 

2:30-5:00 

3 Break-out  groups 

5:00-5:45 

Reconvene  for  whole  group  discussion. 

5:45 

Adjourn  for  Day  1 

4 


June  13 , 2003 


8:00 


8:30 


9:00- 

11:30 

12:45 

1:00 


8:30  A Facility  for  Calibration  of  a 24  m FM  Coherent 
Laser  Radar 

The  presentation  describes  the  existing  facility  used  to 
calibrate  the  MetricVision’s  Coherent  Laser  Radar 
products.  These  products  incorporate  Frequency 
Modulated  Coherent  Laser  Radar,  which  must  be  precise 
and  accurate  over  a wide  range  of  operating 
temperatures.  An  overview  of  the  product  technology, 
and  a review  of  the  calibration  laboratory  and  practices 
will  be  presented. 

9:00  Calibration  Verification  for  Long  Range  Tripod-based 
Terrestrial  Laser  Scanners 

As  with  other  precision  instruments,  terrestrial  laser 
scanners  rely  on  the  manufacturer's  calibration  process 
to  ensure  the  product  meets  specification  at  the  time  of 
shipment.  The  presentation  will  address  the  calibration 
attributes  as  they  pertain  to  long  range  terrestrial  laser 
scanners  range,  and  outline  the  field  procedures  to 
verify  the  calibration  state  of  any  scanner.  The 
procedure  outlined  can  be  adapted  to  serve  the  needs  of 
a wide  variety  of  products  currently  available,  and  is 
based  on  a practical  rather  than  theoretical  approach. 

11:30  3 Break-out  groups 

- 12:45  Reconvene  for  whole  group  discussion 

- 1 :00  Wrap-up  - next  steps 

Adjourn 


D.  Dozor 
Metric  Vision 


E.  Martin 
Optech 
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3. 


SUMMARIES  OF  BREAK-OUT  GROUP  DISCUSSIONS 


In  the  group  discussions,  the  term  “accuracy”  was  frequently  used.  There  were  general 
discussions  of  what  is  meant  by  “accuracy”  and  other  metrology  terms  - hence,  the  call  for 
common  definitions  in  the  workshop  recommendations.  In  the  following  sections,  “accuracy” 
should  be  taken  to  mean  “deviation  from  the  truth”  or  “uncertainty  of  a measurement”  as  these 
interpretations  seems  to  meet  the  intent  of  the  speaker. 

3.1  June  12,2003:  Morning  Session 

The  main  question  for  this  break-out  session  was:  “What  can  we  calibrate?”  The  following 
outline  was  given  to  each  group  as  a guideline  for  their  discussions: 

- current  systems 

o types  of  systems 
o system  characteristics 

o additional  steps  required  to  accommodate  future  systems 
o potential  problems  - system  specific  procedures? 

- what  would  we  like  to  calibrate? 

o range 

o pointing  accuracy  ->  determine  correlations  ->  error  propagation 
o beam  divergence  ->  resolution 

- prioritize 

The  break  out  group  assignments  were  made  to  so  that  each  group  had  a mixture  of  end  users, 
manufacturers,  and  researchers.  This  may  not  have  occurred  due  to  unavoidable  absences. 


3.1.1  Group  1 Summary 


Group  Participants: 

John  Palmateer  (Chair)  Francois  Blais 

Dave  Dozor  Les  Elkins 

Maris  Juberts  Joe  Liadsky 

Nell  Sedransk 


Carlton  Daniel 
Tyler  Estler 
Alan  Lytle 


This  group  considered  four  areas:  types  of  LADARs,  types  of  standards,  measurement 
uncertainties,  and  priorities  for  a national  lab. 

Following  an  initial  review  of  the  types  of  LADAR  systems,  Group  1 then  discussed  the 
characteristics  of  the  various  LADAR  systems  - time-of-flight,  flash,  long-range  low  accuracy, 
short-range  high  accuracy,  etc. 

The  group  then  considered  standards  and  identified  three  types  of  standards: 


7 


1.  Performance  standards  or  commercial  standards.  A fundamental  question  is  how 
does  one  compare  system  A with  system  B and  have  a common  language  so  that  one 
can  make  a rational  decision. 

2.  Customer  uncertainty.  For  an  ultimate  user  of  the  data,  the  xyz  value  is  usually  the 
important  output  from  the  sensor.  Therefore,  the  uncertainty  for  this  data  is  crucial. 

3.  Researcher  and  developer  standards.  These  standards  deal  more  with  specific 
LADAR  system  details  to  try  to  quantify  error  models  and  to  better  understand  and 
improve  performance.  These  standards  are  similar  to  customer  uncertainty  but 
include  more  details  so  that  an  evaluation  of  the  system  is  possible. 

The  group  also  felt  that  standards  were  also  required  for  the  environment  - outdoor,  lab,  factory. 
Seismic  issues  and  ambient  lighting  were  considerations  to  be  taken  into  account. 

Measurement  uncertainty  was  then  discussed.  Potentially  one  needs  to  separate  measurement 
uncertainty  from  analysis/software  uncertainty  (e.g.,  registration  error).  These  uncertainties  are 
usually  combined  and  reported  as  a single  value.  There  may  also  be  other  task  specific 
uncertainties  and  these  tasks  are  often  intrinsic  to  the  numerical  analysis.  The  issue  of 
uncertainties  as  a transfer  function  was  discussed.  Is  this  valid?  What  are  the  issues  related  to 
this? 

Finally,  priorities  for  national  lab  to  assist  users  were  discussed: 

- issue  for  users 

o getting  help  in  setting  performance  standards 
o participants  were  disappointed  that  NIST  does  not  set  standards 
o a lot  of  users  would  like  to  see  performance  standards  developed 

- developers/manufacturers  need  standards 

o environmental  and  seismic  standards 

3.1.2  Group  2 Summary 

Group  participants: 

Don  Channin  (Chair) 

Eric  Martin 
Tony  Slotwinski 
Chris  Witz  gall 

This  group  concentrated  on  two  topics:  calibrations  and  accuracy. 

Currently,  calibrations  are  usually  done  internally  by  the  manufacturer.  Users  have  to  interpret 
these  reported  measurements  in  terms  of  their  requirements  and  have  to  understand  if  a given 


Jim  Albus 
Arkady  Savikovsky 
Bill  Stone 


Eric  Lundberg 
Mike  Shneier 
Nicolas  Vandapel 
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manufacturer’s  system,  as  reported  in  their  specification  sheet,  meet  their  needs.  Users  have 
reported  that  this,  in  general,  is  very  difficult  to  do.  Therefore,  calibration  processes  meant  to 
bridge  this  gap  between  manufacturers’  specifications  and  users’  requirements  are  needed. 

Another  issue  discussed  was  the  measurement  accuracy  of  a point  in  space  (e.g.,  a target  or 
reference).  There  are  absolute  vs.  relative  measurements.  Different  systems  respond  to  different 
targets  in  different  ways.  Therefore,  it  was  recommended  that  an  important  first  step  would  be  to 
develop  a reference  set  of  targets.  The  optics  community  has  standard  gray  scales  and  it  is 
important  to  develop  something  similar  to  this  for  LADAR  systems.  However,  the  development 
of  such  a standard  scale  for  LADARs  is  harder  as  there  is  a wide  range  of  LADARs  - 
instruments  which  use  wavelengths  varying  from  600  nm  to  1500  nm,  high  precision  instruments 
(maximum  range  < 10  m),  factory  scale  (maximum  range  of  about  50  m),  outdoor  scale 
(maximum  range  of  100s  of  meters). 

Environmental  factors  were  identified  as  major  contributors  to  measurement  error,  and  they 
introduce  variability  which  is  difficult  to  control.  For  LADARs  used  outdoors,  the  uncertainties 
of  measurements  will  change  according  to  environment,  especially  in  rugged  environments. 
Specifications  are  usually  obtained  under  a controlled  environment  and  users  need  to  interpret 
how  these  specifications  change  when  used  in  their  work  environment. 

The  group  then  discussed  the  desired  level  of  accuracy  in  a test  facility,  that  is,  < 1 mm,  <10  cm, 
etc.  What  level  of  accuracy  should  a facility  provide?  The  challenge  lies  in  trying  to  deal  with 
different  levels  of  accuracy  - a 1 mm  accuracy  may  be  sufficient  for  one  instrument  but  crude  for 
another  instrument.  This  issue  is  very  difficult  to  solve.  In  terms  of  positional  accuracy,  the 
accurate  distance  between  2 target  points  is  usually  not  sufficient.  It  is  important  to  determine 
what  is  necessary  to  encompass  all  the  different  applications. 


3.1.3  Group  3 Summary 


Group  participants: 


Dave  Gilsinn  (Chair)  Gerry  Cheok 

Joe  Grobmyer  Steve  Hand 

Anders  Ryerson  John  Sandusky 


Chuck  Fronczek 
Dirk  Langer 
Dan  Sawyer 


One  of  the  main  points  made  in  this  group  was  the  need  for  definitions  of  common  terms.  The 
group  could  not  agree  on  the  difference  between  acronyms  LIDAR  (light  distance  and  ranging) 
and  LADAR.  In  army’s  point  of  view,  the  term  LIDAR  was  associated  with  Doppler,  aerosols, 
and  wind  measurements  and  LADAR  was  associated  with  mapping  and  hard  target  detection.  A 
statement  was  made  that  if  all  that  comes  out  of  this  workshop  is  a common  set  of  terminology,  it 
would  be  very  helpful. 


Some  of  the  other  points  discussed  by  the  group  were: 


- standards 
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o hardware  standards 

o software  standards  - a user  may  not  know  what  software  is  used  and  the  results 
would  be  different  for  different  software 
o end  users 

■ want  to  know  how  well  the  system  as  a whole  works 

■ performance  standards  is  preferred  and  more  important  than  hardware 
standards 

o manufacturers 

■ prefer  a set  standard  protocols  and/or  artifacts  which  allow  in-house 
testing  in  lieu  of  a certification  procedure  which  would  involve  shipping 
each  instrument  to  a neutral  facility  as  this  would  be  very  cumbersome  and 
expensive. 

test  facilities 

o many  existing  facilities 
o examples 

■ military  facilities  (Eglin,  Redstone) 

" Canada  (NRC) 

■ private  (Raytheon) 
o How  many  are  out  there? 
o Where  are  they? 

o How  does  NIST  fit  in? 
need  for  a facility 

o from  a manufacturer’s  point  of  view,  set  of  common  procedures  is  needed,  not 
necessarily  a facility 
parameters  currently  being  measured 
o range 
o pointing 
o beam  divergence 
need  to  have  static  and  dynamic  scenes 
some  facility  properties/requirements: 
o include  objects  with  known 

■ dimensions 

■ reflectance 

o need  to  have  targets  which  encompass  the  various  wavelengths  used  by  LADARs 
o positioning  for  pointing 

o types  of  objects  / artifacts  have  to  be  agreed  upon 
o facility  requirements 

■ physical  requirements  will  be  driven  by  application 

■ three  facility  sizes  were  considered 

• 5 m to  20  m 

• 20  m to  50  m (factory  scale) 

• > 100  m (construction) 

what  can  NIST  contribute  - at  the  very  least,  a public  facility 
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3.2  June  12,  2003:  Afternoon  Session 

The  group  assignments  for  this  break-out  session  were  the  same  as  for  the  morning  session.  The 
main  question  for  this  break  out  session  was:  “Select  one  or  two  properties  for  calibration. 
Given  those  properties,  how  would  you  calibrate?”  The  following  outline  was  given  to  each 
group  as  a guideline  for  their  discussions: 

- what  accuracy/resolution  issues  were  encountered? 

o lessons  learned 

- what  are  the  most  important  properties  for  calibration?  for  whom  (manufacturers,  end 
users)? 

- can  we  calibrate  it? 

- how  will  this  change  for  future  LADARs? 

- other  insights 

3.2.1  Group  1 Summary 

The  group  had  wide  ranging  discussions.  The  question  as  posed  by  the  group  chair  was  - “What 
properties  should  be  calibrated?”  If  you  want  to  calibrate  something,  what  are  the  standards  for 
calibration?  There  is  a need  to  define  the  term  “calibration”  and  other  terminology.  For 
example,  one  person’s  calibration  is  another  person’s  compensation  and  it  is  very  important  to 
define  what  is  what.  There  was  a lot  of  discussion  on  the  topic  of  terminology. 

The  group  then  discussed  standards.  The  group  did  not  discuss  facility  requirements  or  what 
standard  the  aircraft  industry  would  look  at  vs.  the  standard  used  by  the  shipping  industry  as 
these  standards  vary  based  on  the  purpose  of  the  calibration.  For  example,  Boeing  uses 
multilateration  technique  with  tracking  interferometers  and  their  standards  are  established 
accordingly.  Earl  Morris  (Redstone  Arsenal)  uses  GPS  for  long  range  (over  kilometers) 
measurements  or  they  use  theodolites. 

Calibration  issues  were  the  next  topic  of  group  discussion.  The  group  felt  that  it  was  very 
important  to  look  at  targets.  What  out  there  are  you  measuring?  They  considered  target  boards 
with  different  targets.  For  diffuse  targets,  the  laser  spot  size  becomes  important.  For  range 
measurements  - point  measurement,  focal  plane  array  - the  target  has  to  be  bigger  than  the  spot 
size.  Therefore,  knowledge  of  the  laser  beam  size  and  characteristics  is  needed.  Likewise,  the 
modulation  transfer  function,  normally  applied  to  cameras,  may  have  utility  for  LADARs. 

The  measurement  distribution  and  density  of  points  for  a given  measurand  was  also  discussed. 
For  example,  for  a given  feature  and  uncertainty,  how  many  points  are  needed  on  that  feature  and 
how  is  the  point  density  related  to  the  resulting  uncertainty.  How  many  points  really  define  a 
circle  and  does  that  change  with  the  uncertainty  of  the  measurements  and  the  placement  of  the 
targets.  This  topic  led  to  a discussion  of  statistical  measurements.  When  reaching  the  limit  of 
what  is  physically  possible,  you  have  to  allow  statistics  to  ferret  out  the  information.  Therefore, 
statistics  need  to  be  part  of  any  calibration  process. 
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Finally,  calibrations  have  to  consider  the  environmental  factors. 


3.2.2  Group  2 Summary 

The  group  discussions  primarily  focused  on  a high  level  (big  picture)  view 

- what  is  the  purpose  the  calibration  facility  and  the  effort  behind  it 

o end  users  - need  less  detailed  data 

o manufacturers  - will  it  be  a unique  facility  where  they  can  test  their  instrument 

- is  it  to  certify  equipment  or  to  calibrate 

o certification 

■ certification,  to  Group  2,  means  that  the  equipment  is  brought  to  a facility 
where  specific  performance  tests  are  conducted.  The  result  is  a 
certification  based  on  the  test  results,  verifying  that  certain  levels  of 
performance  were  achieved.  In  some  cases,  performance  will  be 
determined  by  the  manufacturer  themselves,  e.g.,  if  they  want  their 
equipment  to  measure  a standard  item  to  a certain  degree  of  accuracy.  In 
other  cases,  the  level  of  performance  will  be  set  by  standards,  e.g.,  use  of 
LADARs  in  civil  engineering.  In  that  case,  standards  may  be  established 
for  these  applications. 

o calibration 

■ equipment  specific 

■ difficult  to  find  common  ground  to  calibrate  different  LADAR  systems. 

The  consensus  of  the  group  was  that  the  certification  function  would  be  the  primary  function  of  a 
neutral  facility  with  some  calibration  support. 

The  group  identified  3 scales  for  a facility 

- small 

o 2 m to  3 m range 
o 1 pm  to  1 mm  uncertainty 
o characterized  by  a single  artifact 

- medium 

o 3 m to  20  m range 

o millimeter  to  centimeter  level  uncertainty 
o being  a single  room  environment 

- large 

o > 20  m range 

o 1 cm  to  3 cm  uncertainty,  at  times  may  require  1 mm  at  100  m 
o outdoors 

The  facility  could  also  be  characterized  in  terms  of  the  kind  of  equipment  that  the  LADAR  unit 
might  be  replacing.  For  example,  if  replacing  a CMM  (coordinate  measuring  machine),  a 
medium  scale  facility  could  be  appropriate  and  measurements  using  a laser  tracker  or  other 
surveying  equipment  would  likely  provide  the  required  accuracy.  Consideration  of  how  the 
equipment  is  used  or  what  equipment  is  being  replaced  may  dictate  the  physical  size  of  the 
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operation.  Using  this  criterion,  the  group  identified  some  accuracy  requirements  for  a medium 
size  facility.  For  a medium  size  facility,  there  is  likely  a need  to  measure  down  to  the  level  of  a 
thousandth  of  an  inch  to  be  equivalent  to  a CMM.  For  civil  engineering  measurements, 
measurements  down  to  the  order  of  millimeters  may  be  required  for  capturing  information 
regarding  object  deformation  while  a larger  scale,  on  the  order  of  centimeters,  may  be  acceptable 
for  object  placement.  The  facility  will  have  operational  scale  limits  and  it  was  agreed  that  a 
calibration  facility  will  have  to  have  10  times  the  accuracy  of  the  operational  limits. 

The  group  realized  that  LADAR  equipment  measures  the  pointing  and  ranging  data  (although  the 
end  user  only  uses  xyz  data)  in  different  ways  and  probably  have  different  elements  of  accuracy 
associated  with  each.  This  once  again  raised  the  question  of  calibration  vs.  certification  - if  you 
are  certifying  equipment,  it  is  probably  sufficient  to  get  the  xyz  accuracy  and  not  worry  about 
how  the  ranging  and  pointing  accuracies  contribute  to  the  xyz  accuracy.  But  if  you  want  to 
calibrate  a system,  knowledge  of  component  accuracies  is  very  important.  So  if  the  facility  is 
intended  to  support  calibration,  then  you  have  to  address  this  issue  and  the  fact  that  various 
LADAR  systems  use  different  methods  of  addressing  them.  This  issue  does  not  apply  for 
certification. 

The  group  felt  that  in  addition  to  issuing  a pass/fail  and  certifying  an  instrument  to  a certain 
accuracy,  it  would  also  be  helpful  if  a neutral  facility  included  some  information  as  to  the  source 
of  error.  This  may  or  may  not  be  possible. 


3.2.3  Group  3 Summary 

The  group  looked  at  specifics.  When  calibrating  something,  what  are  you  going  to  do?  Taking 
range  as  an  example,  how  might  you  do  it  here  at  NIST.  Currently  NIST  has  a tape  tunnel  that 
goes  out  to  60  m.  A rail  system  with  a platform  for  targets  positions  the  target  along  the  60  m 
track  and  an  interferometer  is  available  for  use.  There  is  a need  for  standard  targets  with 
multiple  surfaces  and  targets  with  different  reflectances.  A target  board  with  a sub-panel  within 
the  target  that  can  be  moved  in  and  out  of  plane  of  the  target  was  suggested  to  examine  depth 
resolution.  This  test  could  also  be  used  to  study  beam  divergence  - more  blurring  at  the  edges 
would  indicate  a larger  beam  divergence.  A “stair”  target  was  also  suggested  for  studying  depth 
resolution.  A question  was  raised,  “Does  beam  divergence  have  anything  to  do  with  distance?” 
and  the  answer  was  “Yes,  it  does”.  Examination  of  “phantom  points”  or  “mixed  pixels”  also 
requires  some  knowledge  of  beam  divergence.  Targets  for  LADAR  systems  which  record 
multiple  returns  (it  was  felt  that  this  feature  will  become  more  prevalent  in  future  systems) 
should  have  a mesh  in  front  of  the  target  to  pick  up  the  different  return  pulses. 

Reflectivity,  beam  divergence,  pulse  distortion  effects,  and  environment  all  affect  distance 
measurements  and  should  be  considered. 

A question  was  raised  regarding  the  feasibility  of  establishing  a kilometer  range  facility  at  NIST. 
It  was  felt  that  there  was  sufficient  infrastructure  at  NIST  to  establish  such  a facility. 
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3.3  June  13,  2003:  Morning  Session 

The  main  question  for  this  break-out  session  was:  “What  is  the  ideal  facility?”  The  following 
outline  was  given  to  each  group  as  a guideline  for  their  discussions: 

- what  would  an  ideal  facility  for  ground  based  LADARs  do 

o indoor/outdoor 
o modular/unified/consortium 

- what  should  be  required  by  way  of  calibration  to  cover 

o short  and  long  range  systems  (meters  to  kilometers) 
o accuracies  from  microns  to  centimeters 

- select  a component  of  the  ideal  facility  for  a more  detailed  outline 

- should  we  standardize?  can  we  standardize? 

3.3.1  Group  1 Summary 

Eric  Martin  (Chair)  Abdullah  Qassim  Don  Channin 

Chuck  Fronczek  Dave  Gilsinn  Eric  Lundberg 

Tony  Slotwinski  Bill  Stone 

The  discussion  centered  on  “What  is  going  to  come  out  of  this  effort?”  and  on  trying  to  tie  up  a 
lot  of  the  loose  ends.  There  were  a lot  of  ideas  but  there  were  few  common  denominators.  First 
of  all,  the  group  acknowledged  the  fundamental  need  for  an  artifact-based  facility  - irregardless 
of  the  type  instrument.  Rather  than  talk  about  facilities  for  calibrating  devices  (there  was  some 
hesitancy/reluctance  and  legal  connotations  associated  with  the  certification  aspect),  the  group 
talked  about  a neutral  performance  metric  that  would  allow  for  rational  comparison  of  one 
LADAR  against  another. 

The  group  also  talked  about  living  within  the  realities  of  budgets  and  financial  constraints.  It 
was  felt  that  at  present  time,  funding  of  $300  K to  $500  K could  be  available  on  a pilot  project 
basis.  This  amount  would  serve  to  address  some  of  the  short,  medium  and  long  range  systems 
and  to  just  really  “get  our  feet  wet”  - to  begin  addressing  the  neutral  performance  concept  - first 
thoughts  and  impressions  of  what  is  needed  or  not  needed,  before  any  large  commitment  of 
funding  is  made. 

The  types  of  customers  were  discussed.  Who  would  benefit  from  having  this  kind  of  facility? 
The  group  categorized  customers  into  3 classes: 

- 1st  group  - the  really  savvy  leaders  of  the  field  who  really  know  what  they  are  doing 
and  have  intimate  knowledge  of  the  instruments  and  work  with  manufacturers  to 
develop  the  necessary  calibration  specifications 

- 2 group  - those  who  know  what  they  need  but  don’t  necessarily  have  the 
wherewithal  or  a facility  to  test  the  instruments 

- 3rd  group  - naive  customers  who  could  be  persuaded  either  way  with  a sales  pitch  or 
may  not  understand  how  to  compare  and  evaluate  the  different  systems 
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The  group  sees  the  growing  need  for  a facility  simply  because  as  LADARs  get  more  prevalent, 
more  and  more  people  will  use  them.  The  category  of  customers  seen  as  growing  the  most  is  the 
last  category  or  the  naive  customers,  and  the  existence  of  a neutral  facility  then  becomes  all  the 
more  important.  The  group  felt  that  there  was  a need  for  a market  survey  and  the  need  to  include 
others  (besides  those  represented  at  the  workshop)  as  everyone  needs  to  be  on  the  same  page  at 
least  in  the  initial  stages.  The  group  suggested  contacting  other  agencies  to  try  to  avoid 
duplication  of  effort  and  to  try  to  avoid,  more  importantly,  contradictory  information  that  could 
happen  in  a bureaucracy.  Included  in  that  contact  list  are  specific  organizations  that  deal  in  the 
specific  areas  like  some  of  the  survey  groups  - the  metrological  ones  that  have  their  own  unique 
interests. 

In  terms  of  an  action  item  to  move  forward,  the  group  agreed  with  the  need  for  a facility  that 
performs  neutral  performance  measurements/assessments  for  LADARs.  The  classes  of 
performance  are  going  to  vary  and  the  facilities  themselves  would  vary  depending  on  the  type  of 
devices  being  tested.  “It  is  virtually  impossible  to  come  up  with  a single  one  size  fits  all, 
universal,  all  seeing  facility  to  check  standards  that  range  from  microns  to  millimeters  to 
centimeters  over  ranges  that  may  go  out  to  kilometers  - this  simply  becomes  virtually  impossible 
to  do.” 

The  group  discussed  a combination  of  an  indoor  and  outdoor  facility  as  described  in  the 
workshop  presentations.  There  is  also  a need  for  standardized  data  sets  for  software  - for 
software  development  purposes  and  software  analysis. 

In  regards  to  the  question  of  a facility:  A facility  - if  there  is  a need  for  one  and  the  group  has 
argued  that  there  is  - should  NIST  build  it  or  should  someone  else?  This  is  a fundamental  issue 
that  has  to  be  decided  or  at  least  can’t  be  overlooked. 

The  need  to  define  a neutral  performance  metric  was  felt  to  be  a fundamental  consideration  if  a 
metric  is  to  be  tested.  The  metric  needs  to  defined  - what  are  the  important  criteria?  By  having 
this  neutral  performance  metric,  manufacturers  will  be  able  to  standardize  definitions  of 
performance  for  sales  literature,  brochures,  etc.  This  allows  the  3rd  category  of  users  (those  not 
well  versed  in  the  technology  or  in  the  applications)  to  have  a level  playing  field  for  comparison. 
The  group  felt  that  some  progress  was  made  in  this  regard  over  the  last  day  and  a half. 

Also,  there  was  agreement  within  the  group  that  certain  manufacturers  would  certainly  be 
interested  in  an  industry  guide. 

The  group  felt  that  this  initiative  should  move  forward  because  at  least  from  the  manufacturers’ 
and  a lot  of  the  users’  point  of  view,  it  is  something  that  was  seen  as  being  required  as  LADARs 
become  more  prevalent.  The  group  did  not  discuss  detailed  facility  requirements  because  they 
felt  that  it  is  still  too  early  in  the  planning  phase  to  do  so. 
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3.3.2  Group  2 Summary 


Steve  Hand  (Chair)  Gerry  Cheok  Les  Elkins 

John  Sandusky  DirkLanger  Alan  Lytle 

John  Palmateer  Chris  Witzgall 

The  question  of  the  day  was  “What  kind  of  facility  do  these  instruments  need?”  The  first 
comment  was  “Who  is  going  to  operate  it  and  who  is  going  to  use  it?”  Along  with  these 
discussions,  the  group  discussion,  as  occurred  in  the  past  day  and  a half,  centered  on  the  need  for 
common  terminology  or  definitions.  This  need  was  felt  to  be  essential  and  should  be  addressed 
early  in  this  process  of  establishing  a calibration/performance  assessment  facility. 

The  group  discussed  legalistic  issues  regarding  calibrations  vs.  certifications,  the  identification  of 
standards  to  be  tested,  and  the  use  of  the  facility  as  simply  a test  range  with  a third  party  (e.g., 
NIST)  oversight.  These  were  the  different  concepts  of  what  the  facility  would  be  used  for  and 
how  it  would  be  operated.  It  was  felt  that  a facility  would  provide  the  means  for  objective  tests 
and  not  necessarily  a certification. 

The  size  of  the  facility  was  discussed  extensively  by  the  group.  It  was  pointed  out  that  a 2-story 
or  a high  bay  facility  was  needed.  The  group  agreed  that  there  has  to  be  multiple  facilities. 
Having  a single  facility  for  all  types  of  LADARs  is  going  to  be  very  difficult. 

There  was  also  quite  a bit  of  discussion  on  artifacts  and  the  fact  that  the  artifacts  are  going  to 
have  to  be  based  on  the  actual  parameters  (application)  and  how  to  develop  standard  procedures. 
One  topic  during  the  group  discussions  was  the  discussion  of  the  artifacts  being  NIST  traceable. 
What  does  NIST  traceable  mean?  Similar  questions  were  then  raised.  Given  that  such  artifacts 
would  be  available  to  a manufacturer  or  end  user  for  loan  or  rent,  the  group  felt  that  it  was 
necessary  to  develop  procedures  (for  care,  handling,  set-up,  testing,  environmental  test 
conditions)  for  the  use  of  the  artifacts.  The  group  also  discussed  how  end  user  testing  might 
work  in  such  a facility. 

The  group  also  came  to  the  conclusion  that  a facility  is  necessary.  The  group  consciously 
avoided  discussion  of  certification.  They  instead  discussed  the  procedures  for  the  use  of  the 
facility,  the  artifacts,  and  the  maintenance  and  management  the  artifacts  and  the  procedures.  The 
group  felt  that  the  question  of  allocation  of  error  should  not  be  overlooked  and  should  be  dealt 
with,  especially  in  a growing  industry  such  as  LADARs. 


3.3.3  Group  3 Summary 


Dave  Dozor  (Chair)  Francois  Blais  Tyler  Estler 

Joe  Liadsky  Arkady  Savikovsky  Dan  Sawyer 

Nell  Sedransk  Nicolas  Vandapel 

The  group  began  their  discussions  with  “What  does  the  field  look  like  - long  range  systems, 
shorter  range  systems,  and  short  range  low  uncertainty  and  long  range  high  uncertainty  type  of 
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systems?”  The  group  talked  about  different  types  of  facilities  and  the  group  chair  felt  that  at 
least  two  facilities  are  required  - some  others  thought  at  least  three  are  required. 

The  group  felt  that  many  facilities  would  be  required  and  discussed  what  sort  of  overlap  they 
should  have.  The  group  recognized  that  the  most  expensive  facility  would  probably  be  the  small, 
highly  controlled  facility  for  the  short  range  instruments.  On  the  other  hand,  it  may  be  less 
costly  and  actually  advantageous  to  have  many  field-based  facilities  in  different  climates, 
different  terrains  to  help  characterization  of  and  understanding  of  L AD  ARs  by  the  capture  of  the 
entire  scene  rather  than  identifying  tooling  balls  as  is  done  in  industrial-type  metrology. 

What  should  be  characterized?  Error  modes  or  failures?  The  group  discussed  topics  and  areas 
that  required  investigation,  specifically  for  some  of  the  pulsed  laser  radars  - multiple  objects  in 
the  field  of  view,  calibration  for  all  types  of  LADARs,  and  performance 
specifications/verification.  For  laser  radar,  doppler  is  a major  issue  and  therefore,  moving  targets 
are  something  that  should  be  included  in  a test  facility. 

A point  was  brought  up  in  the  group  discussion  concerning  the  relative  size  of  an  organization. 
The  companies  currently  manufacturing  LADARs  are  usually  small  companies  - ranging  from 
20  to  25  people,  definitely,  less  than  50.  Some  companies  have  had  the  advantage  of  working 
with  and  having  the  support  of  much  larger  companies  using  their  instrument.  However,  there 
are  those  companies  which  don’t  have  the  benefit  of  such  technical  and  financial  support.  These 
“companies  that  don’t  have  billions  of  dollars  behind  them  could  really  use  a NIST  facility  to 
help  them”. 

The  group  did  not  discuss  standardization  because  they  thought  that  this  is  a topic  that  required  a 
lot  more  thought  and  time  than  was  available  in  this  workshop.  They  agreed  that  range,  point  to 
point  positioning  accuracy,  different  types  of  materials,  resolutions  of  the  combined  system, 
impact  of  reflectivity  on  measurement,  ambient  lighting  conditions,  relative  humidity, 
temperature,  vibration,  environment  are  all  very  important  parameters  that  are  first  order  for 
study  and  some  standard  procedures  and  methods  need  to  be  developed  to  examine  them.  A lot 
of  these  parameters  extend  across  the  different  types  of  LADARs. 

Drift  tests  and  higher  measurement  resolution  must  be  considered  and  that  means  in  the  entire 
volume.  For  example,  Metric  Vision  measures  many  tooling  balls  all  around  a given  space  but 
they  don’t  segment  the  20  000  m3  to  30  000  m3  space  into  many  1 cm3  spots  and  guarantee  that 
each  point  in  that  volume  as  accurate.  So  the  question  is  where  does  the  laser  radar  fail? 
Another  instance  was  brought  up  whereby  a serious  problem  could  not  be  identified  due  to  the 
lack  of  a sufficiently  large  facility.  The  problem  was  identified  in  the  field  (large  manufacturing 
environment)  and  fixed;  however,  not  before  many  of  the  instruments  were  already 
manufactured.  It  was  felt  that  a NIST  facility,  assumed  to  be  larger  than  what  a small  company 
would  likely  have  or  have  access  to,  could  help  identify  these  problems. 

The  group  talked  about  2.5D  vs.  entire  scene.  If  you  have  a room  full  of  different  types  of 
artifacts,  how  do  you  ensure  that  all  those  things  have  the  point  to  point  accuracy  that  you  want. 
The  2.5D  situation  is  a lot  easier  - you  can  measure  with  a CMM  or  a laser  tracker. 
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Standardization  was  then  discussed.  The  types  of  targets  that  can  be  measured  should  be 
standardized.  The  group  was  resoundingly  positive  on  this  issue.  The  types  of  standards  for 
measuring  range  could  be  standardized  - however,  it  requires  a lot  more  thought.  There  are 
probably  some  other  things  that  can  be  standardized  in  the  area  of  LADARs  - certainly  types  of 
targets  and  range  standards  are  two  that  can  be  addressed  immediately. 
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4. 


WORKSHOP  SUMMARY 


The  Building  and  Fire  Research  Laboratory  of  NIST  conducted  a workshop  as  a first  step 
towards  the  establishment  of  a LADAR  calibration  facility.  The  workshop  was  held  on  June  12- 
13,  2003  at  the  NIST  campus  in  Gaithersburg,  MD,  and  was  attended  by  a representative  cross 
section  of  end  users  and  manufacturers  as  well  as  private  and  government  researchers  from 
Canada  and  the  United  States. 

The  workshop  was  organized  around  six  presentations  and  three  break-out  sessions.  In  each  of 
the  break  out  sessions,  the  participants  were  assigned  to  one  of  three  groups.  After  each  break 
out  session,  all  the  participants  reconvened  and  a summary  of  the  group’s  discussion  was 
presented.  Summaries  of  the  groups’  discussions  are  given  in  Chapter  3.  A general  workshop 
summary  and  future  steps  are  presented  in  this  chapter. 

4.1  Summary 

An  underlying  theme  in  many  of  the  workshop  discussions  was  how  to  deal  with  the 
fundamental  question:  “What  is  calibration?”  More  specifically  “Do  you  really  want  calibration 
or  do  you  want  performance  assessment/evaluation  or  do  you  want  certification?”  The  terms 
“calibration”,  “performance  evaluation”,  and  “certification”  have  similar  meanings  and  have 
been  used,  at  times,  synonymously.  However,  slight  differences  in  the  nuances  of  these  terms 
play  a crucial  role  when  establishing  a facility  for  calibration  or  performance  evaluation  or 
certification. 

It  was  felt  that  calibration  is  performed  to  determine  the  hardware  characteristics  to  enable 
setting  or  alignment  of  instrument  parameters  to  optimal  levels.  A more  formal  definition  of 
calibration  given  by  VIM  [3]  is: 

a set  of  operations  that  establish,  under  specified  conditions,  the  relationship 
between  values  of  quantities  indicated  by  a measuring  instrument  or  measuring 
system,  or  values  represented  by  a material  measure  or  a reference  material,  and 
the  corresponding  values  realized  by  standards. 

Notes: 

1)  The  result  of  a calibration  permits  either  the  assignment  of  values  of 
measurands  to  the  indications  or  the  determination  of  corrections  with  respect 
to  indications 

2)  A calibration  may  also  determine  other  metrological  properties  such  as  the 
effect  of  influence  quantities 

3)  The  result  of  a calibration  may  be  recorded  in  a document,  sometimes  called  a 
calibration  certificate  or  a calibration  report. 

Performance  assessment/evaluation  is  a voluntary  assessment  and  would  be  conducted,  at  the 
request  of  an  end  user,  to  determine  how  well  the  instrument  and  the  processing  software  would 
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meet  the  end  user’s  specific  requirements.  The  performance  assessment  could  also  include 
software  analysis. 


Certification  has  legal  connotations  and  would  involve  testing  of  the  instrument  in  accordance 
with  a set  of  protocols  and  the  results  measured  against  a metric  - pass/fail.  The  testing  would, 
in  general,  be  conducted  in  a certified  laboratory.  Product  certification  is  voluntary;  however, 
lack  of  certification  may  be  interpreted  negatively  - rightly  or  wrongly. 

The  following  example  for  measuring  tapes  is  offered  to  help  clarify  the  difference  between 
certification  and  performance  evaluation  [1]. 

An  American  company  wants  to  sell  measuring  tapes  in  Denmark.  To  do  so,  the  tapes 
have  to  meet  certain  requirements.  They  meet  the  requirements  and  are  certified,  and 
the  company  is  given  the  authority  to  put  the  official  seal  on  their  tapes.  No  individual 
tape  needs  to  be  evaluated  since  they  have  been  certified. 

In  the  U.S.,  the  same  company  simply  sells  the  tapes.  The  customer  either  believes  the 
numbers  or  not.  If  the  accuracy  of  the  tape  is  important,  the  customer  will  request 
traceability  of  the  measurements.  At  this  point,  a higher  authority,  NIST  or  a laboratory 
traceable  to  NIST,  will  be  asked  to  calibrate  the  tape. 

What  then  are  the  needs  of  end  users  and  manufacturers?  Because  of  the  large  investment 
involved  in  acquiring  LADAR  instruments,  users  need  to  have  confidence  in  stated  claims  or 
specifications  and  be  confident  that  what  they  are  purchasing  would  meet  their  particular  needs. 
The  following  measures  would  aid  in  building  this  confidence: 

- clarification  of  manufacturers’  specifications  to  enable  meaningful  comparisons 
between  various  commercially  available  instruments 

- uniform  guidelines  for  manufacturers’  specifications,  testing,  and  reporting 

- performance  testing  of  individual  user-owned  instrument  upon  request  at  a neutral 
facility 

Manufacturers  also  expressed  support  for  the  goals  of  the  workshop.  Although  many  LADAR 
manufacturers  have  gone  to  great  lengths  to  test  and  evaluate  their  products,  they  affirmed  the 
need  for  quality  assurance  and  uniform  specifications  such  as: 

- common  set  of  terminology 

- facilitation  of  “factory  floor”  calibrations  through  the  use  of  NIST  traceable 
artifacts  and  standard  procedures 

- availability  to  manufacturers  of  a climate  controlled  facility  for  testing/calibration, 
particularly,  under  extreme  conditions 

- uniformity  of  specification  testing  and  reporting 

The  LADAR  output  of  main  importance  to  most  users  is  the  xyz  data.  However,  as  the  LADAR 
output  is  typically  a large  point  cloud,  processing  methods  are  to  be  included  in  the  testing 
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process  to  provide  “end”  or  “total”  performance  evaluation.  For  manufacturers,  however, 
accurate  information  of  the  hardware  performance  is  essential  for  instrument  improvement. 

Another  difference  between  end  users  and  manufacturers  is  the  apparent  need  for  a facility.  For 
end  users,  a neutral  facility  where  one  may  send  an  instrument  for  performance  evaluation  is 
desirable.  On  the  other  hand,  the  majority  of  manufacturers  at  the  workshop  prefer  a set  of 
standard  protocols  and/or  artifacts  which  allow  in-house  testing  over  a certification  procedure. 
This  was  because  certification  would  involve  shipping  each  instrument  to  a neutral  facility  and 
this  would  be  very  cumbersome  and  expensive.  Properties  of  interest  to  both  users  and 
manufacturers  include  range,  beam  pointing,  beam  size/spread,  and  processing  multiple  returns 
(mixed  pixels  or  phantom  points). 

The  general  consensus  was  that  a single  facility  that  would  encompass  the  entire  range  of 
LADARs  would  be  impossible.  Therefore,  three  kinds  of  testing  facilities  were  envisioned  as 
being  necessary: 

- a small,  highly  climate  controlled  indoor  facility  for  highly  accurate,  short  range 
instruments  (<  10m). 

- a medium  sized,  climate  controlled  indoor  facility  for  instruments  with  ranges  up 
to  50  m. 

- an  outdoor  testing  area  for  long  range  instruments  and  for  testing  in  a more 
realistic  environment. 

While  the  emphasis  at  the  workshop  was  on  ground-based  LADARs,  the  outdoor  facility  could 
be  extended  for  use  for  airborne  LADARs.  There  was  also  an  opinion  that  input  from  the 
airborne  LAD AR  community  should  be  sought  in  this  “standardization”  process,  at  least  during 
the  early  stages,  as  there  were  similarities  between  the  ground-based  and  airborne  instruments. 

The  issue  of  “Why  have  standards?”  was  covered  in  the  presentations  and  in  the  break-out  group 
discussions.  The  participants  felt  that  standards  would: 

- provide  a means  for  uniform  performance  evaluation.  As  the  use  of  LADAR 
grows  and  there  are  more  “naive”  or  nascent  end  users,  the  ability  to  fairly 
compare  systems  is  invaluable.  Similarly,  when  contracting  for  LADAR  services, 
the  ability  to  insert  performance  standards  into  contracts  is  very  helpful. 

- allow  end  users  to  do  conduct  their  primary  business,  i.e.,  manufacture  planes, 
build  rail  systems,  and  not  have  to  undertake  the  task  of  designing 
calibration/testing  procedures  and  protocols.  Having  to  devote  personnel  to  this 
task  is  costly  and  often  financially  difficult  for  smaller  companies. 

In  general,  there  was  strong  support  for  standardization.  In  fact,  a specific  request  for  LADAR 
standards  was  made  to  the  then  NBS  (National  Bureau  of  Standards,  now  NIST)  as  far  back  as 
the  early  1980s.  However,  it  was  recognized  that  standardization  involves  a long,  arduous,  and 
sometimes  tortuous  process.  It  was  pointed  out  that  the  standardization  of  a process  requires  the 
implementation  of  proof-of-concept.  This  would  be  a potential  NIST  role. 
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In  summary,  the  applications  for  LADARs  are  seen  to  be  growing  rapidly.  These  applications 
include  commercial  automation,  urban  planning,  mapping,  surveying,  autonomous  vehicle 
navigation,  global  climatology  monitoring,  bathymetry  (measurement  of  water  depth),  and 
homeland  security  (possibly  for  chemical  and  biological  weapon  detection).  This  being  the  case, 
the  need  for  some  neutral  facility  (whether  for  performance  assessment  or  calibration  is  yet  to 
decided)  was  almost  universally  agreed  upon.  There  were  three  common  themes  that  ran 
throughout  and  stood  out  in  the  discussions.  These  recurring  themes  centered  on  the  need  for: 

- common  set  of  terminology 

- standard  targets/artifacts/standard  reflectivity 

- performance  assessment/evaluation 

It  was  suggested  that  a good  starting  point  for  developing  definitions  for  common  terminology  is 
the  VIM  [3]  and  the  Guide  to  the  Expression  of  Uncertainty  Measurement  or  the  GUM  [2]. 


4.2  Future  Steps 

Of  the  three  common  issues  listed  above  was  one  that  could  and  should  be  addressed 
immediately:  the  need  for  a set  of  definitions  of  common  terms  for  LADARs.  Therefore,  it  was 
suggested  that  NIST  initiate  the  creation  of  a set  of  common  definitions,  addressing  in  particular, 
accuracy/precision/resolution  to  be  sent  to  the  participants  for  comment. 

In  addition,  the  following  steps  were  also  suggested: 

- contact  other  professional  organizations  for  possible  collaboration/coordination 

- conduct  a review/inventory/benchmarking  of  existing  facilities 

- publish  a definition  of  terms  or  characteristics  of  LADAR  systems  to  include 
similarities  and/or  differences  of  systems  (a  survey  of  commercially  available 
instruments  was  published  by  POB  magazine  - www.pobonline.com) 

- create  a list  of  standard  targets  and  range  standards.  Possibly  conduct  a survey. 
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APPENDIX  A:  LIST  OF  WORKSHOP  PARTICIPANTS 


Dr.  Qassim  Abdullah 


Dr.  Jim  Albus 


Capt.  Jon  Bailey* 


Francois  Blais 


Bruce  Borchardt 


Don  Channin 


Gerry  Cheok 


Carlton  J.  Daniel 


EarthData 

7320  Executive  Way 
Frederick,  MD  21704 
qabdullah@earthdata.com 

NIST 

100  Bureau  Drive,  Stop  8230 
Gaithersburg,  MD  20899-8230 
j ames . albus@nist.  go  v 

National  Geodetic  Survey 
N/Ngs3  SSMC3 
1315  East  West  Highway 
Silver  Spring,  MD  20910 
j on.bailey@noaa.gov 

National  Research  Council  of  Canada 
M-50,  1200  Montreal  Road 
Ottawa,  Ontario  K1A  0R6 
Canada 

ffancois.blais@nrc-cnrc.gc.ca 

NIST 

100  Bureau  Drive,  Stop  8211 
Gaithersburg,  MD  20899-8211 
bruce.borchardt@nist.gov 

Samoff  Corporation 
201  Washington  Road 
CN5300 

Princeton,  NJ  08543-5300 
dchannin@samoff.com 

NIST 

100  Bureau  Drive,  Stop  8611 
Gaithersburg,  MD  20899-8611 
cheok@nist.gov 

U.S.  Corps  of  Engineers 
Topographic  Engineering  Center 
7701  Telegraph  Road 
Alexandria,  VA  22315-3864 


(301)  948-8550 


(301)  975-3418 


Not  available 


(613)  993-3285 


(301)  975-3469 


(609)  734-2383 


(301)  975-6074 


(703)  428-6766  x 2273 
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Bob  Dillon* 


Dave  Dozor 


Les  Elkins 


Nick  England* 


Dr.  Tyler  Estler 


Charles  Fronczek 


Joe  Grobmyer 


Dr.  David  Gilsinn 


Robert  Habbit* 


carlton.j.daniel@erdc.usace.army.mil 

Lasermet  Systems,  Inc.  (781)  273-5995 

103  Terrace  Hall  Avenue 
Burlington,  MA  01803 
bob.dillon@LumenLabs.com 

MetricVision  (703)  550-2945  x 126 

8500  Cinder  Bed  Road 
Suite  150 

Newington,  V A 22122 
dave_dozor@metricvision.com 

Spatial  Integrated  Systems  (301)  610-7965  xl07 

7564  Standish  Place,  Suite  100 
Rockville,  MD 
lelkins@sisinc.org 

3rdTech 
PO  Box  248 
Chapel  Hill  NC  27514 
nick@3rdtech.com 

NIST 

100  Bureau  Drive,  Stop  8211 
Gaithersburg,  MD  20899  - 821 1 
tyler.estler@nist.gov 

NIST 

100  Bureau  Drive,  Stop  8211 
Gaithersburg,  MD  20899  - 821 1 
charles.fronczek@nist.gov 

U.S.  Army  Aviation  and  Missile  Command  (256)  876-1094 
Aviation  & Missile  Research, 

Development,  and  Engineering  Center 

AMSAM-RD-MG-IR 

joe.grobmyer@rdec.redstone.army.mil 

NIST  (301)975-3491 

100  Bureau  Drive,  Stop  8910 
Gaithersburg,  MD  20899  - 8910 
david.gilsinn@nist.gov 

Sandia  National  Laboratories  (505)  845-9492 

P.O.  Box  5800 


(919)  929-1903 


(301)975-3483 


(301)  975-4079 
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Steve  Hand 


Dr.  Tsai  Hong 


Geoffrey  Jacobs* 


Maris  Juberts 


Dr.  Gary  Kamerman 


Dr.  Dennis  Killinger* 


Ted  Knaak* 


Richard  Laboy* 


Albuquerque,  NM  87185-0329 
rdhabbi@sandia.gov 

MagLevInc 

Tech  One  Office  and  Research  Center 
2000  Tech  Center  Drive 
Monroeville,  PA  15146 
shand@maglevinc.com 

NIST 

100  Bureau  Drive,  Stop  8230 
Gaithersburg,  MD  20899-8230 
hongt@nist.gov 

Cyra  Technologies,  Inc 
4550  Norris  Canyon  Road 
San  Ramon,  CA  94583 
geoffrey.jacobs@cyra.com 

NIST 

100  Bureau  Drive 
Gaithersburg,  MD  20899  - 8230 
maris.juberts@nist.gov 

FastMetrix,  Inc 
P.O.  Box  14208 
Huntsville,  AL  35815 
gwk@ladar.org 

University  of  South  Florida 
Dept,  of  Physics 
University  of  South  Florida 
Tampa,  FL  33620 
killinge@chuma.cas.usf.edu 

Riegl  USA 

7081  Grand  National  Drive,  Suite  112 
Orlando,  FL  32819 
tknaak@rieglusa.com 

Boeing 

P.O  Box  3707,  Mail  Stop  5H-77 
Seattle,  WA  98124-2207 
richard.  laboy  @boeing . com 


(412)  816-1200x6097 


(301)  975-3444 


(925)  790-2317 


(301)975-3424 


(256)  881-5558 


(813)  974-3995 


(407)  248-9927 


Not  available 
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Dr.  Jay  Land* 


Dr.  Dirk  Langer 


Joe  Liadsky 


Eric  Lundberg 


Alan  Lytle 


Dr.  Raj  Madhavan 


Eric  Martin 


Earl  Morris 


Air  Force  Research  Lab,  Eglin  (850)  882-1724  x 104 

Laser  Radar  Research  Facility 

Air  Force  Research  Laboratory  Munitions  Directorate 

Advanced  Guidance  Division 

101  W Eglin  Blvd,  Suite  219 

Eglin  Air  Force  Base,  FL  32542 

jay.land@eglin.af.mil 

Z&F  (412)469-9210 

One  Library  Place,  #203 
Pittsburgh,  PA  15110 
langer@zf-usa.com 

Optech  (416)661-5904 

100  Wildcat  Road 
Toronto,  Ontario 
Canada  M3J  2Z9 
j oe@optech.  on.  ca 

Lundberg  Consulting  Not  available 

1470  Park  Garden  Lane 
Reston,  VA  20194 
elundberg@mac.com 


NIST  (301)975-6048 

100  Bureau  Drive,  Stop  861 1 
Gaithersburg,  MD  20899  - 861 1 
alan.lytle@nist.gov 


NIST  (301)  975-2865 

100  Bureau  Drive,  Stop  8230 
Gaithersburg,  MD  20899  - 8230 
madhavan@nist.gov 

Optech  (416)  661-5904 

100  Wildcat  Road 

Toronto,  Ontario 

Canada  M3J  2Z9 

eric@optech.  on.  ca 

U.S.  Army  Aviation  and  Missile  Command  (256)  876-1094 
Aviation  & Missile  Research, 

Development,  and  Engineering  Center 

AMSAM-RD-MG-IR 

earl.morris@rdec.redstone.army.mil 
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John  Palmateer 


(253)  931-4636 


Richard  Richmond* 


Anders  Ryerson 


Dr.  Kamel  Saidi 


Dr.  John  Sandusky 


Arkady  Savikovsky 


Dan  Sawyer 


Dr.  Nell  Sedransk 


Mike  Shneier 


Boeing 

P.O  Box  3707,  Mail  Stop  5H-77 
Seattle,  WA  98124-2207 
john.w.palmateer@boeing.com 

Air  Force  Research  Lab 
richard.richmond@wpafb.af.mil 

National  Geodetic  Survey 
1315  East  West  Highway 
Silver  Spring,  MD  20910 
Anders.Ryerson@noaa.gov 

NIST  - BFRL  - CMAG 
100  Bureau  Drive,  Stop  8611 
Gaithersburg,  MD  20899  - 86 1 1 
kamel.saidi@nist.gov 

Sandia  National  Laboratories 
MS  0329 
PO  Box  5800 

Albuquerque,  NM  87185-0329 
jvsandu@sandia.gov 

Laser  Projection  Technologies 
8 Delta  Drive 
Londonderry,  NH  03053 
arkady  @LPT  corp . com 

NIST 

100  Bureau  Drive,  Stop  8211 
Gaithersburg,  MD  20899-8211 
daniel.sawyer@nist.gov 

NIST 

100  Bureau  Drive,  Stop  8980 
Gaithersburg,  MD  20899-8980 
nell.sedransk@nist.gov 

NIST 

100  Bureau  Drive,  Stop  8230 
Gaithersburg,  MD  20899-8230 
michael.shneier@nist.gov 


(937)  255-9614x250 

Not  available 

(301)  975-6069 

(505)  845  -0132 

(603)  421-0209x238 

(301)  975-5863 

(301)  975-2853 

(301)975-3421 
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Tony  Slotwinski 


Dr.  Bill  Stone 


Nicolas  Vandapel 


Bryan  Williams* 


Dr.  Christoph  Witzgall 


Metric  Vision 
8500  Cinder  Bed  Road 
Suite  150 

Newington,  V A 22122 
tony_slotwinski@metricvision.com 

NIST 

100  Bureau  Drive,  Stop  8611 
Gaithersburg,  MD  20899  - 861 1 
william.stone@nist.gov 

Carnegie  Mellon  University 

Robotics  Institute 

NSH 

5000  Forbes  Avenue 
Pittsburgh,  PA  15213 
vandapel@ri.cmu.edu 

Mensi 

35  Technology  Parkway  South 
Suite  170 

Norcross,  GA  30092 
bryan.  williams@mensi.  com 

NIST 

100  Bureau  Drive,  Stop  8910 
Gaithersburg,  MD  20899  - 8910 
christoph.witzgall@nist.gov 


(703)  550-2945 


(301)  975-6075 


(412)  268-2991 


(770)613-5255 


(301)  975-2725 


* Was  not  able  to  attend  workshop  but  requested  to  remain  on  the  mailing  list  for  future 
activities. 
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APPENDIX  B:  LADARATNIST 


by  William  Stone 
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Fanning  Laser 

Automation  Group 


Non-Line-of-Sight  Metrology 
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Why  is  it  imp^efi^J 


Within  ten  years  will  almost  completely  replace 
traditional 


It  vvllliBTiable  complete,  real-time^D___^ 
representatiomlof  complex,  unstructured 
manufacturing^ scene#and  permit  automation  to 
take  place  within^these  regimes. 


Manufacturing 

Inspection/Automation 


Present  LADAR  Use:  Topo  Maps: 

Military  & Civilian 

Construction:  As-Builts  apps. 


Urban  Planning 
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Digital  FIRM 


Present  LADAR  Use 

Bathymetry 
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Automated  Steel  Construction 


■■■- 


Construction  Metrology  & Automation  Group 


NIST  Role 

• Provide  Industry  Forum  for 
Design  of  Innovative  Steel 
Connections 

• Provide  neutral  testing  lab 
for  evaluation  of  novel 
connections 

• Provide  Robotic  Testbed  for 
Design  and  Performance 
Evaluation  of  automated 
placement  systems 

• Explore  radical  automati 
architectures  for  steel 
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LADAR-machine  visio 
auto-docking  trajector 


HHSH  I Hi 


esses  4D  site 
sequencing  and 
e “seed”  target 
nal  placement 
via  real-time 
ior  lock. 


PIMS 
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HMMHBgHIHVl 


Present  Real-time  State-of-Art  for  FPA  LADARs  (30  Hz+  x 32  x 32  pixel) 


Construction  Metrology  & Automation  Group 


Courtesy  Riegl  USA  and  Lincoln  Labs 
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DARPA  Urban  Autonomous  Navigation: 
Sensor-Fused  Signage  Recognition 
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CMAG:  Advanced  LADAR  Research  (OA) 


40 


DARPA  JIGSAW: 

Seeing  through  Clutter 
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CMAG:  Advanced  LADAR  Research  (OA) 
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Implications  for  construction:  auto-generatio 
of  bare  terrain  models;  auto-ID  of  machinery! 
Fast  removal  of  clutter  from  as-builts. 
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METROLOGY 
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MEASURES 
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Analysis  of  Real-World  LADAR  Data:  High-Res  HMMWV  Scan 
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FY02:  Locate  Space  and  Implement  Prototype  Facility 

to  begin  end-to-end  metrology  performance  delivery 


Indoor  LADAR 
Test  Facility 


Environmentally  controlled 
Range  calibration  - 1 pm  to  0.1  mm  accuracy 
Pointing  accuracy  - 0.001  deg.  accuracy 
Artifacts 

• Size -dimensions  accurate  to  0.1  mm 

* Shapes  - to  be  determined 

♦ Location  of  artifacts  - accurate  to  0.1  mm  position:  0.01  deg  orientation  control 
Determination  of  accuracy  of  3D  model  - inherent  in  this  value  is  instrument 
accuracy,  registration  accuracy,  and  accuracy  of  surface  generation  algorithm 


Construction  Metrology  & Automation  Group 


Outdoor  Facilit 


* Enable  gr 
generatioi 


Evaluate  different 
systems  for  mobility 
applications 


Enable  characterization 


of  individual  sensor 
(hardware  & software) 

Generate  calibrated 


data  for  evaluation 
of  perception 
algorithms 


Enable  transfer  of 
calibration 

procedures  - secondary 
and  traveling  standards 


Constructor.  Metrology  <i  Aote.YUjtfc.il  Group 
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APPENDIX  C:  CALIBRATION  ACTIVITIES  AT  THE  NATIONAL 
RESEARCH  COUNCIL  OF  CANADA 


by  Francois  Blais 


NIST:  LADAR  Calibration  Workshop 
June  12  - 13,  2003 


National  Institute  of  Standards  and  Technology,  Gaithersburg,  MD 


Calibration  Activities  at  the  National 
Research  Council  of  Canada 


1 pi 


Frangois  Blais 


National  Research  Council  of  Canada 


Ottawa,  Ontario,  Canada 
Francois.Blais@nrc-cnrc.gc.  ca 
www. vit . iit .nrc . ca 


CMC  ARC 


Outline 


■ The  NRC  - National  Research  Council  of  Canada 

■ Active  3D  Techniques  - examples 

■ Calibration 

• Modelling  and  Calibration  procedure 

• Practical  considerations  - methodology 

• Field  Calibration 

• Assessing  the  accuracy 

■ Conclusion:  the  need  for  standards 


Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2002  GNRC'KiC  * 


49 


G NRC  - A National  Institution 


Calibration  Activities  at  the  NRC 


NIST  Calibration  Workshop  2003  CfctC't&C  2 


Visual  Information  Technology  Group 
Research  activities 

* Virtualizing  reality  and  visualization 

• 3D  laser  sensors  - metrology 

• Shape  and  colour  modeling 

• Modeling  from  2D  images 

• Advanced  visualization  systems 

* 3D  data  mining  & management 

• 3D  Search  Engine 

• Advanced  interfaces  for  3D  searching 

* Collaborative  Virtual  Environments  (CVE) 

Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CKtC'NPtC  3 
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□ 3D  Acquisition  Systems  at  NRC 

■ 3D  sensor  development 

• Active  triangulation  (Biris  + Auto-synchronized) 

• Range:  10  cm  to  20  m 

• Accuracy:  10  um  to  a few  cm 

• Data:  range  + registered  intensity  (color) 

■ Calibration 

• 3D  Geometrical  (LUT,  model  based,  hybrid) 

• Color  reflectance 

• Accuracy  assessment:  Triangulation,  Pattern  Projection  , TOF 

■ Processing  software 

• ICP,  Segmentation,  model  reconstruction,  characterisation 

• Perspective  projection  (2D->3D) 

■ Applications 


Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  4 


H Dimensional  inspection  - Modeling 


3D  pattern  projection  and  photogram metrv 


Calibration  Activities  at  the  NRC 
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Active  3D  Techniques  at  NRC 
and  some  examples 


Measuring  3D  shape:  Light  waves 


Light  waves 
f=  100  to  1000  THz 


Holographic 
Speckle  or  White 
light- based 


Focuslde-focus:  Confoca!  microscopy 


see  also  D.  Nitzan  PAMI,  v.10,  n.3,  1988 


Calibration  Activities  at  the  NRC 
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□ Auto-synchronization  : 3D  + colour 


Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CfirlC'fiAC  9 
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13  High-resolution  scanner  for  3D  & colour 


Simultaneous  colour  & range  digitizing 

• high  resolution:  az  ~ 10  fjrri  , Ax  ~ 50  jjm 

• Small  volumes  10-20  cm 

• Intrinsic  reflectance  measurement 


Calibration  Activities  at  the  NRC 
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□ From  Reality  to  Virtualized  Reality 


Calibration  Activities  at  the  NRC 
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From  Reality  to  Virtualized  Reality 
(movie) — 


Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CtfiC'NtC  12 
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□ Large  Field  of  View  Laser  Scanner 


o 

© 

o 


monochrome  system  (532  nm,  820  nm,  1.5  urn) 
dual-axis  scanning  galvanometer  mirrors 
for  digitizing  large  structures 
operates  on  a conventional  tripod  or  telescopic  tripod 


• range  50  cm  to  10  m,  resolution:  70  pm  at  50  cm 


Calibration  Activities  at  the  NRC 
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Mission  STS-105  International  Space  Station 

August  2001 


Neptec  Design  Group  (under  license) 
Collaboration  Neptec-NASA-CSA-NRC 


Photo  NASA 


Calibration  Activities  at  the  NRC 
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H Reflectance  and  Shape  Information 


■ Mission  STS-105 


Photo  Neptec 


Calibration  Activities  at  the  NRC 
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Tracking  Moving  Targets 

during  MPLM  Demate 


■ Produces  real-time  pose  estimates  for  assembling  the  ISS 
elements 


Courtesy  Neptec 


Calibration  Activities  at  the  NRC 
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■ Sensor  Calibration 


■ LUT  based  calibration  (1986) 

• [x  y zj  =[LUTx[p,0,<p]  LUTy{p,dM  LUTz{p,0,<p]Y 

• Large  memory  tables 

• Not  practical  for  large  volumes 

■ Hybrid  Calibration  (1988-90) 

• [x  y zj  = \fic(z,6,(p)  jy(z,0,<p)  fz(p-,[6,cp])J 

• LUT  of  simplified  models 

« Model  based  calibration  (1992s  - 2000s) 

• [x  y zf  =fip,0,(p) 

• Strong  astigmatism  + high  correlation  in  XYZ 

• Mechanical  model:  non-linear  (unstable) 

• Optical  “virtual”  model  + observability:  (stable) 

\u  v.  w]r  -f(p, 0,(p)  [x  y z]r=[«/w  v/w  l/w]J 

Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  Cf^tC  KtC  ^ 


■i  Model  Based  Calibration 


■ Lens  model  does  not  apply 

• Strong  astigmatism 

• Non-linear  correlation 
between  rotation  axis  / 

• Scanning  mirrors:  non- 
symmetric  distortions 

• Distortions  vary  with 
range  Z 

« Minimization  of  errors: 

• High  correlation  in  XYZ 

• Origin  of  the  scanner  *«■ 

(x0,y0,Zo)  undefined 


Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CNrtC'NrtC  ^0 
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■ I 


Calibration 


Calibrating  a Laser  Scanner 
Practical  considerations 


■p 

(0.4  m to  20  m) 


Metrology  laboratory 


Calibration  Activities  at  the  NRC 
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■ Example  of  Equipment 

■ Temperature  controlled  environment 

■ Anti-vibration  tables 

■ Theodolites  and  photogrammetry  equipment 

■ Optical  measurement  and  alignment  tools  (e.g 
interferometers) 

■ High  precision  reference  objects 

» Optical  quality  reference  surfaces 

■ Accurate  linear  and  rotation  stages 

• Inspection  software  (e.g.  Polyworks) 

■ Access  to  high  precision  CMM  and  other  measuring 
capabilities  through  other  Institute  and  partners 

■ 
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■ Calibration  Experimental  Set-up  (old) 


■ Structure  originally  used  for 
model  calibration 

• Require  accurate 
survey  of  the  targets 

■ Complexity 

• Cost 

• Can’t  be  used  for 
simultaneous  calibration  of 
small  and  large  volumes 

• Small  number  of  targets 
(accuracy) 

• Occlusions  and  obstructions 
■ Good  to  verify  the  calibration  not  to  calibrate  scanners 

Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CNrtC'NrtC  24 
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H Calibration  with  a Target  Array 


■ 


Known  positions 

Accurately  machined 
target  array 

Range  limited  by  the 
translation  stage 

Does  not  cover  the  full 
FOV 

Wobble,  jitter,  backlash, 
orthogonal  motions 

• E.g.  20  urad  wobble  = 
200  urn  XYZ  error 

A mechanical  “challenge” 


Calibration  Activities  at  the  NRC 
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High  Precision  Calibration  using 
T argetArray  ’ 

■ Space  Vision  System  (NASA) 


Photo  Neptec 


Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  Gt'pttZ'KTC'  26 


62 


Acquiring  Calibration  Data 


Calibration  Activities  at  the  NRC 


NIST  Calibration 


entente 


■ As  many  point  as  possible 

■ Automated:  to  reduce  user  errors 


Geometrical 

tracking 


Lissajous 

Imaging 


Raster  Imaging 
+ ROI 


H How  can  we  solve  for  the 
“Absolute  Reference”  ? 

■ Several  XYZ  coordinate  systems:  one  for  each 
reference  object 

■ Must  accurately  measure  the  absolute  position  of  the 
camera  vs.  these  reference  systems 

* All  6 DOF  (translation+rotation) 

• Camera  optical  reference  : a hidden  “virtual”  point 
inside  the  camera  mechanical  enclosure 

♦ Camera  enclosure  (assembly  tolerance) 

• Absolute  location  for  each  target  array,  object,  or 
structures  used  during  the  calibration 


Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CNrtG’NrtC  20 
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■ Example:  Calibration  Procedure 


■ Field  calibration 

■ Large  number  of 
measurement  = better 
statistics 

■ Redundancy  = detect 
misaligned  references 

■ Semi-Automated  = 
reduced  user  errors 

■ Fast  procedure 

■ User  friendly 
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Assessing  the  Accuracy 
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□ Verification 

■ Using  independent  methods  and/or  reference 
object 

• Different  calibration  plates 

• Reference  objects  acquired  from  different 
orientations  + differential  inspection 

• Using  another  3D  scanner 

• Other  systems  and  methods 

- Photogrammetry,  theodolites,  CMM 


Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CNOC'NrtC 


□ Calibrate,  verify  & compare 


Calibration  Activities  at  the  NRC 


NIST  Calibration  Workshop  2003 


CKRCMC 
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■ Uncertainty:  align  2 overlapping  scans 


Ax,y=0.05mm 

Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CfcrtC 't&C  33 


Laser  scanner  #2 


Fringe  projection  #2 


Ax,y=0.2mm 


Calibration 

m 

The  need  for  standards 

;X: 


UBi 
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□ What  means  accurate  3D  ? 

■ Single  point,  surface  deviation,  or  edge 

■ Single  view  or  reconstructed  object 

■ Accuracy  vs.  spatial  resolution  and  volume  (zmin,zmax,z2) 

■ Processing  software  rarely  considered  when  assessing 
accuracy 

• Noise  reduction:  l / Vw~ 

- Fit,  smoothing,  dithering,  and  time  averaging 

- Affects  spatial  resolution  (blur) 

* May  not  necessarily  increase  accuracy 

- Correlated  XYZ,  bias 

- Can  induce  distortion, and  shifts  (bias) 


Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CSrtC'NrtC  3 5 


□ About  standards 

■ Definitions:  VIM  1993 

■ Theodolites:  section  3,  German  standard 
DIN  18723 

■ CMM  manufacturers  use  ISO  standard 
10360-2:1994  for  assessing  their  measuring 
machines 

■ Guideline  called  VDI/VDE  2634  is  being 
prepared  in  Germany  for  optical  3-D  vision 
systems 

• testing  and  monitoring  procedures 

• area  scanning 

Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CKtC’hRC  36 
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■ Certification:  NIST,  NRC  ... 

■ Procedures 

■ Certification  2D  or  3D:  NIST  (US),  NRC 
(CAD),  ISO  (Italia),  PTB  (DE),  ASME  B89.4.1- 
1997  ... 

■ For  3D  systems  based  upon  pattern 
projection:  VDIA/DE  2634  (DE) 

■ Definitions  of  specifications:  Vocabulaire 
International  de  Metrologie  (VIM) 

• Resolution,  Uncertainty,  Accuracy 
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■ Conclusion 

■ Calibration  is  a complex  iterative  process 

• Understanding  the  scanner  / model 

• Calibration  methodology,  equipment,  reference 

• Independent  accuracy  assessment 

* Experience  shows  that  many  errors  are  often  related 
to  the  reference  object 

• Assumption  that  the  reference  object  is  accurate 

• Machining  tolerance 

■ Calibration  procedure  must  be  carefully  followed 

• Still  very  prone  to  human  errors,  manipulation,  and 
procedural  errors 

Calibration  Activities  at  the  NRC  NIST  Calibration  Workshop  2003  CNrtC'SrtC  ^8 
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■ Conclusion  (coni) 

■ Need  “good”  standards  to  avoid  misleading  “claims” 

• Current  comparisons  are  based  solely  on 
“specifications”  from  manufacturers  ... 

• Different  standards  are  needed,  adapted  for  a 
given  type  of  application 

- Edges,  surface,  or  closed  models 

- Data  smoothness  vs.  spatial  resolution 

- The  non-linear  properties  of  3D  scanners 

- Operating  conditions 
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■ T erminology  ( 1 ) - from  VI M 

* Accuracy:  closeness  of  the  agreement  between  the  result  of  a 
measurement  and  a true  value  of  the  measurand.  If  one  wishes  to 

measure  absolute  quantities  then  this  is  important. 

■ Reproducibility:  closeness  of  the  agreement  between  the 
results  of  measurements  of  the  same  measurand  carried  out 
under  changed  conditions  of  measurement.  One  needs  this 
feature  if  the  same  artefact  has  to  be  measured,  let  say,  at 
different  times  or  by  a different  user. 

* Uncertainty  characterizes  the  dispersion  of  the  values  that 
could  reasonably  be  attributed  to  the  measurand.  The 
measurement  uncertainty  can  be  further  decomposed  in  a 
systematic  and  a random  part,  (d) 
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□ Terminology  (2) 

■ Systematic  errors:  mean  that  would  result  from  an  infinite 
number  of  measurements  of  the  same  measurand  carried  out 
under  repeatability  conditions  minus  a true  value  of  the 
measurand.  This  type  of  error  can  be  due  to  poor  calibration . 
range  measurement  artefact . ambient  light  conditions , 
reflectance  properties  of  surface  etc.  Can  be  reduced  by 
modeling  the  errors. 

* Random  errors:  this  type  of  error  originates  from  stochastic 
temporal  and  spatial  variations  of  influence  quantities.  One  has 
to  lower  this  quantity  if  nice  synthetic  shadings  of  surfaces  are 
required  without  resorting  to  excessive  filtering. 

■ Resolution:  smallest  difference  between  indications  that  can 
be  meaningfully  distinguished.  One  needs  this  feature  in  order 
to  avoid  being  limited  by  quantification  noise , CCD  resolution , 
spot  size , etc. 

Calibration  ActMties  at  the  NRC  NIST  Calibration  Workshop  2003  CNRC'NiC  41 
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APPENDIX  D:  FUTURE  OF  LASER  RADAR 


by  Gary  Kamerman 
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(Geiger  mode) 

Intensity  Proportionate  to  Square  of  Electric  Field 

• Sensitive  Element  Responds  to  Incident  Intensity 

• Signal  Current  Proportionate  to  Incident  Intensity  in  Linear  Mode 

• Signal  Current  Constant  after  First  Photon  in  Geiger  Mode 

• No  Polarization  Dependence  (w/o  external  elements) 

) 2003,  FastMetrix,  Inc. 
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e Extraction 


Velocity  by  Doppler  & 


M— 


Angular  Position 


a 

B \ 

D J 

(. A + B)-(C  + D ) 
A+B+C+D 
(A  + C)  — (B  + D) 
A+B+C+D 


) 2003.  FastMetrix,  Inc. 
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mSm 


itantaneous  Field  of  View 

. + -C  „ • 1 „ J „1, 


♦ Field  of  View  (FOV)  is  the  total  angular  extent  of  t 
receiver  detector  array  in  a staring  system  or  the  total 
angular  coverage  of  the  receiver  detector  element(s)  during 
a complete  frame  cycle  for  a scanned  system 

♦ Field  of  Regard  (FOR)  is  the  total  angular  extent  over 
which  the  FOV  may  be  positioned 


5 2003,  FaslMetrix.  Inc. 
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TiMsmxTmxmrnmwmmmMMmwm' 


Modulation  Technique 


Direct 


Not  Applicable 


Coherent 


Heterodyne 
Homodyne 
Offset  Homodyne 
Three  Frequency 
Heterodyne 


Pulsed 

AM 

FM 

Hybrid 
None  (C\ 


Functions 

Tracking 

Moving  Target  Indication  (MTi) 
Machine  Vision 
Velocimetry 
Wind  Shear  Detection 


Measurements 


Target  Identification 


Amplitude 

Range 

Angular  Position 
Velocity 

Vibration  Spectra 
Imaging 


<&  2003.  FastMetrix,  Inc. 
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- Air-to-Air  Missile  Laser  Radar  Seeker 

- AM/CW  C02  Heterodyne  Optical  Radar 

- CW  C02  Homodyne  Doppler  Wind  Lidar 

♦ Broad  Based  Research 

- ERIM,  NASA  MSFC,  MIT/LL,  BMDO,  Army  (NVL) 
Air  Force  (ATL),  Navy  (China  Lake) 


')  2003.  FastMetrix.  Inc. 


78 


Fringe  Pattern 
Focal  Region 

Particle  in  focal  region 
scatters  light  into  receiver 

Transverse  motion  moves 
particle  from  fringe  to  fringe 

Rate  of  change  of  modulated  signal  is  a measure  of  velocity 
Radial  motion  does  not  modulate  signal 
Near  field  operation  only 


02003,  FastMetrix,  Inc. 
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♦ Particle  in  beam  scatters  light  into  receive 

♦ Radial  motion  Doppler  shifts  scattered  radiatio 

♦ Optical  heterodyne  mixing  generates  mo 
Doppler  frequency  in  receiver 

♦ Doppler  frequency  is  a measure  of  velocity 

♦ Near  or  far  field  operation 


© 2003,  FastMetrix,  Inc. 
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35000  35500  36000  36500  37000  37500  38000  38500 


Time  after  trigger  {microseconds) 


— Ambient  Conditions 

♦ High  Performance 

- 5 km/sec  maximum  velocity 

- 1 m/sec  velocity  accuracy 

, - 25  |_isec  time  resolution 

- 1500  millisec  event  duration 


©2003,  FastMetrix,  Inc. 


Pulse  C02  Transverse  Excitation  Atmosphej 


0.6  to  1 Joule  per  Pulse 

• 20  Hertz  Pulse  Repetition  Frequency 
Heterodyne  Receiver 

• liquid  N2  cooled  HgCdTe  Detector 

• 20  centimeter  clear  aperture 


© 2003.  FastMeirix,  Inc. 
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mMSUm-.  ■ ■ 


Altitude  (km)  Altitude  (km) 


MMM 


mmmmi 


MACAWS  (AIRBORNE  DOPPLER  LIDAR) 


* to4 

Absolute  Backscatter  fm^sr1) 


LOS  Velocity  (ms*1) 
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mmM 


mBm 

■EM 


Imaging 

Optics 
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Courtesy  of  Optech 
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Height  Above  Ground  Level 


Geometrically  Corrected  Intensity 

© 2003,  FastMetrix,  Inc. 
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n 


■ m 


Railroad  Museum,  Baltimore 

Courtesy  of  Optech  and  the  U.S.  Armv 


Ravens  Stadium,  Baltimore 
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!r& -ef&fot) 


rroposphere 




Lower  Troposphere 

— 


Surface 


10-11  io-10  io-9  io-8  10-7  io-6  to-5  10-4  10-3 

Rackscatter  (m^sr1)  _ , fV(C1 

v Courtesv  of  NASA-MSFC 
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FastMetrL 


♦ Independent 

- Small  Cross 

- Isotropic  Sc 

♦ Cross  Section  scales  with 

♦ Distinctive  Line  Structure  based  upon  Vibra 
Rotational  Energy  States  of  Molecular  Specie 

- water  vapor  & ozone 

- Temperature 

- optical  extinction 

- optical  backscatter 


€>  2003,  FastMetrix,  Inc. 


Courtesy  of  Dr.  Russell  Philbrick,  Pennsylvania  State  University 
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(b)  Hepatocellular  s 

C'  a Dr'rv7i'\^f  a 2 


(b)  Colon  Adenocarcinoma 


Carcinoma 


(a)  Normal  Liver 


(a)  Normal  Colon 


(b)  Renal  Ceil  Carcinoma 


RAMAN  SHIFT  (< 


RAMAN  SHIFT  (cm-*) 


spectrosopic  characterization  of  human 


Douglas  C Redd,  Christopher  J.  Frank, 


malignant  tissues:  implications  for  a 


percutaneous  optical  biopsy  technique 


for  m-situ  tissue  Diagnosis,  ’ Optical 


Biopsy , SPIE  Press,  VoL  2081,  pp.  185 


(a)  Normal  Kidney 


1200 


1400 


1000 


RAMAN  SHIFT  las’1) 


♦ 406,7  nm  Excitation 

* 5 cm*1  resolution 
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♦ Lambertian  Convention 

- Cross-sectional  area  of  a perfectly  reflecting  flat  Lambertian 
oriented  normal  to  the  line  of  sight  from  the  transmitter,  that 
produces  a reflected  energy  density  at  the  laser  radar  receiver  equal 
to  that  produced  by  the  target 

♦ Isotropic  Convention 

- Cross-sectional  area  of  a perfectly  reflecting  flat  isotropic  surface, 
oriented  normal  to  the  transmitter,  that  produces  a reflected  energy 
density  at  the  laser  radar  receiver  equal  to  that  produced  by  the  target 
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Notes 


Cube  Comer 
Reflector 


Flat  Lambertian  Disk 


Long  Wires 
Extended 

Lambertian  Surface 


i Volumetric  Scatterer 


Raman 


ntpV'ArP 

4 

^TRNAnp2r2 


z = sphere,  disk  or  wire  radius  (m) 
l = length  of  wire  or  cube  edge  (m) 


= Raman  cross  section  (cm'  / sr  / molecule), 
N = density  of  Raman  species  (molecules  / m3), 

Ar  — range  resolution  of  the  Iidar  (m)  « ~—s 

p = total  hemispherical  reflectance 
(1  = aerosol  backscatter  coefficient  (m  1 ) 


■ 
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'ALIBER  III  (obsolete) 

Faceted  Target  Surface  Model 

- Uniform  Illumination 
(Continuous  Plane  Waves) 

- No  Shadowing 


- Faceted 

- Arbitral 
(Ray  G 

- Shadow 
— Single  { 

♦ DELTAS  (I 

- Contini 


Arbitra 
(Ray  C 
Shadov 
Multiple  Scattering 
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Cube  Comer  @ 1.06  microns 
Cube  Comer  @ 10.6  microns 
Sphere 
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♦ Modulation  Transfer  Function  (MTF) 

♦ Point  Spread  Function 

♦ MTF  Area  (MTF  A) 

♦ Square  Root  Integral 

♦ National  Imagery  Interpretability  Rating 
Scale  (NIIRS) 
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M = modulation 

Lmax  = maximum  image  luminance 
Lmm  = minimum  image  luminance 
co,  u = vertical  and  horizontal  spatial  frequencies 
oj  = output  and  input,  respectively 
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-2.31616x 

0.20000x11 


Spatial  Frequency 


grog 


♦ Empirically  Deri1 

- Human  Monochromatic  Response 

- Young  Eyes 

- >18  inches 
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Metrics 

Precisio 

Resoluti 

Accurac 


Laser  Cross  Section 


♦ Targets 

- Specular  Reflectivity 

- Diffuse  (Lambertian)  Refelctivity 

- Geometric  Targets 

— Chemical  and  Aerosol  Cross  Sections 

♦ Image  Quality 
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APPENDIX  E.l:  LADAR  MEASUREMENTS  A T BOEING 


by  John  Palmateer 


Ladar  Measurements  at  Boeing 

John  Palmateer 
Technical  Fellow,  Metrology 
Boeing  Commercial  Airplanes 


Boeing  High  Tech  Center 


• Developed  both  AM  and  FM  ladar 

- Worked  with  Digital  Optronics,  now  MetricVision 


• Boeing  continues  work  on  AM 

- Boeing  Laser  Tracker  (BLT) 

• 1 1 systems  used  for  737  assembly 

• Comer  cubes  required 

- Doppler  velocimeter 

• Non  contact,  fast 

- Currently  adapting  velocimeter  for  ranging  up  to  20ft 
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Applications  at  Boeing 


• Testing  of  BLT  and 
Trackers 

FARO  (SMX)  Tracker 
Leica  Tracker 

BLT 

HP  Differential  Interferometer 


JEFMWJEF* WJhfMSr 

Applications  at  Boeing 


• Assembly  Jigs 
on  tracks 

• Adjust  dihedral 
and  sweepback 
relative  to  floor! 

• Not  shown 

- Plumb  bob 

- Pogo  stick 

- Come-along 
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Applications  at  Boeing 


• 737  Wing-Body  Join 

- Simplified  tooling  for 
final  assembly  using 
determinate  assembly 

- Optimizes  dihedral 
and  sweepback  of 
wings  relative  to 
fuselage 


^J5r£JJEYAf£Z 

Application  at  Boeing 


• 737  Body  Join  in 
Wichita 

- Measure  sub- 
assembly 

- Re-measure 
during  assembly 
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FIA-18  Inner  Wing  Applications 


m 


Scan  Wing  Substructure 
Metallic  Features 

& 


MV200 

Laser 

Radar 
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b 

Determine  Actual  Skin  Thickness 
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I* 

w*  3^  a;  ✓ SSSSt  * 
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* I ft  jgHg  ' 

> JPP^v  ^ ..JW 


System  has  passed  acceptance  testing  and  is  qualified  for 
production  development 
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Application  at  Boeing 


• Low  accuracy  ladar 

- Measurement  of 
fuselage,  dorsal  fm,  and 
wings 

- Separately  fit  fuselage, 
and  wings 

- Minimize  asymmetry 


;g^J&£7JFS/V£7 

Application  at  Boeing 


Low  accuracy  ladar 

- Establish  symmetry 
during  final  assembly 

- Average  over  surfaces 

• Standard  deviation  of 
the  mean  “improves” 
raw  uncertainty 

• Perturbation  studies 
indicate  ~0.01  degree 

- Reverse  Engineering  of 
facilities 
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Early  Calibration  Effort 


• Examine  calibration  in  large  volume 

- Use  tracker  as  standard 

- Use  t-Test  for  estimate  of  significant  error 

• Concern  over  relative  uncertainties  between 
standard  and  CLR100 

• Simple  way  to  guardband 


*1  - *2  ~ 
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Early  Calibration  Effort 


CLR  Uncertainty 
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Significant  pj-pj 
between  CLR  and 
Tracker 


Early  version  of  the  T-Test  Computation 
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Early  Calibration  Effort  Sill 
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• The  primary  concern  was  that  the  range  error  would  be  non-linear 

• Discontinuity  between  three  local  oscillators 
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Recent  Calibration  Effort 


• More  experience  with  ladar  measurements 

- BLT 

- Trackers  with  ADM 

• ASME  B89.4.17  - Performance  Evaluation  of 
Laser  Based  Spherical  Coordinate  Measurement 
Systems 

• Effort  to  simplify  calibration 

- Eliminate  need  for  moving  equipment  to  calibration  lab 

- Reduce  support  equipment  costs 


Recent  Calibration  Effort 


• Calibration  of  transverse  measurements  in 
the  near  field 

- Standard  can  create  large  angles  (reduce 
common  mode  errors) 

- Near  field  includes  lateral  and  angular  errors 
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Recent  Calibration  Effort 
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Recent  Calibration  Effort 


• Range  calibration 

- Many  different  ranging  schemes  with  different 
modes  of  failure 

• Two  calibration  methods  being  examined 

- Using  tracking  interferometer  as  standard 

- Using  invariant  distances  in  conjunction  with 
known  distance 
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Recent  Calibration  Effort 
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Comer  Cube 
(move  and  measure 
at  12  places 
on  LOS  between 
instruments) 
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Recent  Calibration  Effort 


<  30  ft 

*  24  ft 

<  20  ft 

<  14  ft > 

*  10  ft > 


LR200  Toolmg  Balls  Certified  Scale  Bar 

(5  places) 


• Method  2 (being  tested)  - Invariant  distance  with 
absolute  scale 

• Change  position  of  ladar  and  look  for  changes  on 
intervals 

- Hope  non-linearity  moves  from  one  interval  to  another 

- Maintaining  invariance  is  a problem 
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Recent  Calibration  Effort 


• Shop  floor  compensation  / calibration 

- Increase  equipment  utilization 

- Increased  uncertainty  balance  with 
compensation  in  measurement  environment 

• Reduced  thermal-mechanical  errors 

• Calibration  of  measurements  as  used 

• Separate  measurement  calibration  from 
software  accreditation 


Conclusion 


• 3D  non-contact  measurement  using  ladar  is  a quantum  leap 
ahead  of  triangulation  and  vision  systems 

- More  expensive 

• Current  challenges  include 

- System  architecture  - CAD  driven  measurement,  analysis  and 
reporting 

• A calibration  standard  is  an  important  step  that  needs  doing 
now 

• NIST  as  a neutral  clearing  house  for  ladar  technology 

- Error  budgets 

- Modes  of  failure 

• NIST  continuing  to  organize  technology  workshops 
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APPENDIX  E.2 


Ladar  Measurements  at  Boeing 

John  Palmateer 
Technical  Fellow,  Metrology 
Boeing  Commercial  Airplanes 
John.  W.Palmateer@Boeing.com 

253-931-463  6 


Abstract: 

The  use  of  laser  radar  (ladar)  measurement  at  Boeing  is  minimal.  Nonetheless,  Boeing  perceives 
the  need  for  non-contact  ladar  measurement  and  has  been  funding  research  toward  this  goal  since 
the  late  1980s.  The  current  efforts  in  ladar  development  have  been  directed  toward  systems 
capable  of  measuring  parts  and  assemblies  in  a manufacturing  environment.  Goals  include  on- 
machine  measurement  and  measurement  during  assembly  for  control  and  inspection.  Calibration 
is  performed  in  a shop  environment.  Calibration  compares  a measurement  to  a length  standard, 
whereby  both  range  and  angle  measurements  are  examined  as  a system.  Because  of  the  less 
certain  nature  of  ladar  (e.g.,  varying  modes  of  failure),  the  range  device  is  also  calibrated 
separately.  Shop  practices  also  include  steps  that  help  to  validate  the  measurement  process,  such 
as  closure  measurement  and  measurement  / verification  of  a length  standard.  This  paper 
envisions  a national  laboratory  with  expertise  in  various  ladar  systems,  which  will  ensure  a better 
understanding  of  the  modes  of  ladar  failure. 

Introduction 

The  extent  of  ladar  measurements  at  Boeing  is  minimal,  so  in  that  respect  the  title  is  misleading. 
Nonetheless,  Boeing  perceives  the  need  for  non-contact  laser  radar  measurements  and  has  been 
funding  research  toward  this  goal  since  the  late  1980.  Initial  efforts  centered  at  the  Boeing  High 
Tech  Center,  and  examined  both  AM  and  FM  systems.  Through  those  efforts  and  in  conjunction 
a partner,  Digital  Optronics  which  has  since  morphed  into  MetricVision,  two  systems  evolved: 
the  Boeing  Laser  Tracker  (BLT),  an  AM  laser  radar  requiring  cooperative  target  to  achieve 
adequate  signal  to  noise;  and  the  MetricVision’ s CLR100,  an  FM  ladar  capable  of  measuring 
directly  off  surfaces.  The  current  errors  in  laser  radar  have  been  oriented  toward  laser  radar 
measurement  systems  capable  of  measuring  parts  and  assemblies  in  a manufacturing 
environment.  Since  that  time,  both  systems  have  been  evolving  and  improving.  Boeing  has 
since  developed  a high  speed,  lower  accuracy  laser  radar  pointed  by  galvanometers  and 
accoustoptics  that  can  track  cooperative  targets  on  rapidly  moving  objects  (e.g.,  a vehicle 
aligning  for  refueling).  We  have  also  used  laser  radar  for  mapping  of  vibrations  (measurement 
of  Doppler).  MetricVision  likewise  has  continued  development  of  laser  ladar  including  an  array 
system  for  inspection  of  surfaces  in  hazardous  environments  (inside  nuclear  waste  storage  tanks). 

Applications 

The  initial  application  for  ladar  in  commercial  airplanes  was  wing-body  join  (WBJ)  for  737.  The 
goal  was  to  eliminate  the  large  floor  assembly  jigs  that  “precisely”  hold  the  orientation  of  the 
wings  to  the  body  during  the  join  operation.  This  was  to  be  replaced  with  a jacking  system  on 
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the  fuselage  and  wings,  whose  relative  orientation  was  controlled  by  measurement  of  those 
assemblies.  Prior  to  the  advent  of  ladar,  this  task  was  not  feasible  because  Boeing  would  have 
needed  to  use  tracking  interferometers.  The  requirement  for  carrying  a retro-reflector  from 
point-to-point  and  the  likelihood  of  breaking  the  laser  beam  was  very  high,  so  use  in  a 
production  process  was  not  feasible.  The  BLT  and  CLR100  were  both  prime  candidates  for  this 
measurement  task.  Ultimately  the  CLR  was  selected  for  the  737  wing-body  join.  The  BLT  was 
later  selected  for  737  body-body  join  in  Wichita.  One  of  the  crucial  (and  less  than  satisfactory) 
aspects  for  both  these  applications  has  been  seamless  integration  of  measurement,  controls  and 
CAD.  This  continues  as  a hot  topic  within  Boeing. 


Figure  1.  Testing  of  Boeing  Laser  Tracker,  and  other  Tracking  Interferometers 

Recently  Boeing  has  applied  ladar  measurements  to  the  FA- 18  (St.  Louis)  and  Wedge  Tail 
(Seattle).  Again  the  need  was  for  a targetless  point  and  measure  system  was  deemed  important. 
For  the  FA- 18  application,  the  need  is  to  measure  the  wing  thickness  at  all  drill  locations  on  the 
wing  for  drilling  of  blind  holes.  The  wing  thickness  is  affected  by  liquid  shim  material  and 
variance  in  the  graphite.  WEDGE  TAIL. 
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Figure  2.  Wing  Substructure 


Calibration 


A major  concern  during  the  development  and  application  of  the  CLR100  to  the  737  WBJ  was  the 
allocation  of  calibration  risk  between  Boeing  and  MetricVision.  Confidence  in  the  CLR100  had 
not  developed.  The  compensation  processes  for  both  the  pointing  head  and  ranging  system  were 
being  developed  and  proven.  From  Boeing’s  perspective  the  calibration  requirement  looked  at 
measurements  accuracy  and  reliability  in  a large  volume  as  would  be  required  during  WBJ. 
MetricVision  did  not  want  the  anomalous  measurements,  typical  in  a development  project, 
disqualifying  the  system,  and  more  important,  creeping  criteria  changing  the  measurement 
requirements  and  costing  extra  time  and  development  dollars.  Furthermore,  as  part  of  a purchase 
contract,  these  concerns  had  to  balance  before  the  system  was  ready  for  prime  time  and  fix  those 
criteria  in  the  contract  writing. 

The  standard  for  the  calibration  was  the  tracking  interferometer.1  Targets  in  a 40  meter  volume 
were  mapped  using  measurements  from  at  least  four  tracking  interferometer  positions.  These 
were  “bundled”  using  Boeing  written  software.  A Student-T  criteria,  using  the  computed 
standard  deviation  from  the  tracker  measurements  and  estimated  uncertainty  of  the  CLR100 
(15  ppm)  to  balance  the  uncertainties  between  the  tracking  interferometer  and  CLR100. 
Additionally,  certain  number  of  measurements  needed  to  be  within  the  accuracy  requirements. 
The  acceptance  criteria  did  not  look  at  the  angle  encoding  accuracy.  Acceptance  criteria  did  not 
explicitly  examine  range  accuracy,  though  it  did  get  examined  separately  on  several  occasions. 

The  most  recent  calibration  document  (handout)  is  very  different  from  the  prior  concept.  First, 
Boeing  has  more  experience  and  confidence  with  ladar  system  (e.g.,  the  Leica  and  Faro  tracking 
interferometers  with  absolute  distance  measurement).  Second,  the  ASME  B89.4.19, 


1 Palmateer,  J.,  “Uncertainty  Characterization  for  Portable  Variable  Geometry  Coordinate  Measurement  Systems”, 
Proceedings  8th  International  Metrology  Congress.  October  1997,  Bordeaux,  FRANCE. 
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Performance  Evaluation  of  Laser  Based  Spherical  Coordinate  Measurement  Systems,  has  been 
looking  at  calibration  issues  and  procedures  related  to  tracking  interferometers.  Discussions 
between  developers,  users  and  interested  scientists  have  resulted  in  several  key  ideas:  transverse 
calibration  in  the  near  field  when  characterizing  measurement  uncertainty;  and  calibration  of  the 
ranging  system  since  these  systems  vary  greatly  and  are  probably  the  least  well  known  and  stable 
aspect  of  the  measurement  system.  Third,  is  an  effort  to  get  the  calibration  process  out  of 
calibration  labs  and  into  the  factory.  This  increases  equipment  utilization  time  and  makes  the 
measurement  closer  to  a process  specific  calibration. 
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APPENDIX  E.3 


Calibration  of  Structured  Light  Metrology  Systems 

John.W.Palmateer@Boeing.com 

1.0  INTRODUCTION 

This  document  outlines  a generalized  procedure  for  calibrating1  structured  light  (a.k.a. 
Moire)  metrology  systems.  This  calibration  may  also  apply  to  other  metrology  systems 
that  perform  non-contact  surface  measurements  such  as  laser  and  scanning-laser  spot 
triangulation.  It  is  anticipated  that  calibration  procedures  for  specific  systems  will  be 
written  based  on  tests  and  criteria  herein  and  as  well  as  other  operational  requirements.  It 
is  strongly  recommended  that  a performance  evaluation  be  performed  for  any  system  and 
application.  The  performance  evaluation  should  include  tests  not  normally  found  in  a 
calibration  related  to  fitness  of  use  such  as  the  ability  to  discriminate  between  good  and 
bad  data,  and  various  methods  that  assist  the  operator  in  this  assessment. 

The  calibration  uses  a ball  plate  artifact  (see  Appendix  E.3-A)  and  optionally  a ball  bar. 
The  artifact  is  made  of  several  (e.g.,  20)  precision  balls  held  in  stable  (i.e.,  three-point) 
mounts.  The  inclusion  of  the  ball  bar  is  designed  to  augment  the  artifact  in  the  event  that 
it  is  substantially  smaller  than  (less  than  50  %)  the  field  of  view  of  the  system  under  test. 
The  ball  bar,  when  used,  should  span  approximately  80  % of  the  system  under  test’s  field 
of  view.  The  calibration  of  measurements  must  be  separated  from  calibration  of  analysis 
routines  that  fit  measured  objects.  When  calibrating  a complex  object  using  CAD  it 
becomes  difficult  to  distinguish  the  measurement  uncertainty,  CAD  model  uncertainty, 
and  the  uncertainty  in  the  fitting  process.  Thus  this  calibration  routine  uses  spheres 
whose  analysis  is  well-understood  and  published. 

This  calibration  is  from  a single  vantage  point.  It  estimates  the  uncertainty  for  the  basic 
unit  of  measurement  (a.k.a.,  X-,  Y-,  Z-triplet,  the  measurement  triplet,  or  voxel)  and  the 
distortion  across  the  field-of-view  (FOV).  One  must  distinguish  between  the  FOV  of  the 
camera  and  projector,  a 2-D  space,  and  the  measurement  volume,  a 3-D  space. 

1 ISO  International  Vocabulary  of  Basic  and  General  Terms  defines  calibration  as  comparison  to  a standard. 
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Distortions  in  the  2-D  space  create  measurement  errors  in  the  3-D  space.  It  is  important 
to  fill  the  FOV  of  the  camera  and  projector  system  so  that  lens  distortions  are  maximized 
in  the  3-D  volume.  In  general,  metrology  systems  where  physical  movement  of  the 
metrology  sensor  is  not  required,  measurement  of  the  artifact  in  a portion  of  the  volume 
captures  the  entire  volumetric  measurement  uncertainty.  That  is,  projection  and 
reconstruction  of  structured  light  extend  via  lines-of-sight  through  the  measurement 
volume.  Thus  the  uncertainty  is  not  strong  function  of  the  volume.  So  unless  there  is 
some  mechanism  that  interacts  with  the  measurement  volume  (e.g.,  focus  mechanism  or 
gantry  movement  of  the  sensor),  absence  of  distortion  in  one  FOV  of  the  volume  implies 
absence  of  distortion  throughout  the  volume.  Thus  calibration  can  use  a “flat”  artifact, 
such  as  a ball  plate,  that  fills  the  FOV  but  does  not  fill  the  measurement  volume.2 3  If  the 
sensor  requires  movement  by  a gantry  or  similar  device,  as  in  the  case  of  some  line  scan 
cameras,  then  the  uncertainty  of  motion  through  the  volume  must  be  characterized  and 
added  to  the  sensor  uncertainty. 

A typical  measurement  of  a feature  is  often  constructed  from  many  individual  X-,  Y-,  Z- 
triplets.  In  the  case  of  this  standard,  the  feature  is  the  center  of  a sphere.  Clearly,  the 
process  of  fitting  X-,  Y-,  Z-triplets  to  complex  surfaces  and  determining  that  fitting’s 
confidence  interval  will  be  a complicated  function  of  X-,  Y-and  Z-triplet  locations,  triplet 
uncertainty,  data  density,  and  surface  contour.  The  role  played  by  triplet  uncertainty  is 
typical  of  any  value  that  is  being  averaged,  in  that  it  influences  the  uncertainty  of  that 
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4 


Thus,  while  the  triplet  uncertainty  primarily 


influences  the  uncertainty  of  a feature  being  extracted,  increasing  the  number  of 
measurements  (as  well  as  other  geometry  considerations)  ameliorates  this  influence. 
Thus,  the  triplet  uncertainty  must  be  known  for  subsequent  uses  of  the  data  but  is  not  part 
of  the  systems  immutable  uncertainty. 


2 Shakaiji,  C.,  “Least-Square  Fitting  Algorithms  of  the  NIST  Algorithm  Testing  Service”,  J.  Res.  Natl.  Inst. 
Stand.  Technol.  103,  633  (1998). 

3 Calibrations  using  “flat”  artifacts:  J.  Zhang  and  A.  Djordjevich,  Study  on  laser  strip  sensor,  Sensors  and 
Actuators  (72)  1999;  K.  Harding,  Calibration  methods  for  3D  measurement  systems,  Proc  of  SPIE,  Vol., 
4189  (2001);  G.  Sansoni,  M.  Carocci  & R.  Rodella,  Calibration  and  performance  evaluation  of  a 3D 
imaging  sensor  based  on  the  projection  of  structured  light,  IEEE  Trans,  on  Instrumentation  and 
Measurement.  Vol.  49,  No.  3 (2000) 
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The  distortion  across  the  FOV  is  not  a value  that  is  reduces  with  additional 
measurements.  Rather,  the  distortion  across  the  FOV  affects  the  measurements  between 
features  in  the  usual  manner  that  is  expected  of  measurement  uncertainty.  The  single 
vantage  point  estimates  the  uncertainty  of  a point-to-point  measurement  via  comparison 
to  a ball  plate.  Thus,  the  triplet  uncertainty  is  independent  of  the  instrument  uncertainty, 
and  not  explicitly  in  its  uncertainty  computation.  The  triplet  uncertainty  is  contained  in 
the  location  in  the  spheres,  which  in  turn  estimate  the  distortion  across  the  FOV. 
Increasing  the  number  of  measurement  reduces  the  distortion  only  because  the  estimates 
for  the  sphere  centers  are  improved.  Thus,  the  distortion  across  the  FOV  is  the  true  point- 
to-point  instrument  uncertainty. 

As  the  volume  is  expanded,  the  uncertainty  becomes  the  RSS  of  the  point-to-point 
uncertainty  plus  the  mechanism  used  to  expand  the  volume.  In  this  manner,  the 
uncertainty  is  propagated  when  combining  images  into  a larger  survey.  For  example,  if 
photogrammetry  is  used  to  stitch  vantage  points  together,  the  resulting  uncertainty  is  a 
combination  of  the  point-to-point  uncertainty  (Upt.pt),  as  determined  by  this  calibration 

process,  and  the  photogrammetry  uncertainty  (Uphotogrammeoy):  • 

Additionally  the  grid  of  photogrammetry  points  throughout  the  survey  also  serves  as  the 
check  for  distortion  as  the  data  is  subsequently  manipulated.  An  alternative  example  is  a 
line  scan  camera  mounted  on  a certified  gantry  where  the  uncertainty  of  individual  line 
scans  are  adjusted  by  the  uncertainty  of  the  coordinate  frame  of  that  certified  gantry. 
Note  that  in  this  case,  the  pointing  of  the  gantry’s  coordinate  frame  should  be  the 
dominant  contributor.  Once  determined,  the  uncertainty  can  be  propagated  into 
application  software  when  generating  surfaces,  extracting  features,  etc. 

Thus,  the  single  vantage  point  calibration  allows  comparison  of  similar  equipment.  It  is 
likely  that  different  products  will  have  different  procedures  for  collecting,  evaluating  and 
combining  data,  so  use  of  a single  vantage  point  provides  a common  denominator 
between  all  processes  so  that  any  system’s  fundamental  unit  of  measure  can  be  compared. 


4 Box,  G.E.P.,  Hunter,  G.W.  and  Hunter,  J.S.,  Statistics  for  Experimenters,  John  Wiley  & Sons  (1978). 
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Procedures  for  calibrating  a fixed  structured  light  installation  that  includes  a translation  or 
rotation  stage  should  also  include  these  translation  and  rotation  stages  when  designing  a 
calibration  procedure.  It  is  relatively  simple  to  modify  the  single  vantage  point  procedure 
for  multiple  vantage  points  and  obtain  an  uncertainty  for  the  fixed  installation.  When  the 
structured  light  system  is  used  in  an  unstructured  manner,  that  is,  combining  vantage 
points  is  ad-hoc , it  may  not  be  practical  to  include  the  mechanism  for  combining  the 
vantage  points  into  the  calibration  procedure.  It  is  strongly  recommended  that  the 
mechanism  for  combining  vantage  points  be  included  as  part  of  the  performance 
evaluation  for  the  application,  but  not  the  calibration  of  the  structured  light  system  since 
it  is  easier  and  more  versatile  to  handle  these  calibrations  separately. 

2.0  SURVEY 

The  single  vantage  point  for  the  calibration  plate  should  be  off  a comer  of  the  plate  and 
diagonally  across  the  plate  (as  shown  in  the  figure  A1  in  Appendix  E.3-A).  The  sample 
size  for  any  object  in  the  survey,  (e.g.,  sphere  on  a ball  plate)  must  include  at  least  100 
measurement  points.  If  the  calibration  plate  subtends  less  than  50  % of  the  metrology 
system’s  field  of  view,  supplement  the  calibration  with  a ball  bar  whose  length  subtends 
at  least  80  % of  the  metrology  system’s  field  of  view.  Additionally,  if  the  depth  of  field 
for  the  metrology  system  is  sufficiently  large  that  it  generates  a significant  change  in 
uncertainty,  then  measurements  of  the  calibration  plate  at  several  positions  within  the 
field  of  view  are  required.  Often  the  depth  of  field  is  small  and  this  is  not  an  issue. 
Alternatively,  the  depth  of  field  may  be  large,  but  only  the  worst  case  uncertainty  is 
desired,  so  only  one  measurement  in  that  worst  case  region  should  be  required.  If  this 
process  rejects  any  data,  the  rationale  for  rejection  must  be  documented  including 
whether  data  rejection  is  automated  or  operator  invoked. 

3.0  ANALYSIS 

The  measuring  and  fitting  of  single  spheres  shows  errors  which  estimates  the  X-,  Y-  and 
Z-triplet  uncertainty.  Measurements  between  independent  balls  quantify  the  distortion 
across  the  field  of  view.  These  two  values  are  estimated  separately,  the  former  for  use  in 
estimating  uncertainty  in  feature  extraction  and  the  latter  for  estimating  the  point-to-point 
measurement  uncertainty. 
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Compute  a sphere  fit  with  unconstrained  radius  for  all  points  on  each  hemisphere  on  the 
ball  plate  (see  Appendix  E.3-B).  Record  the  computed  radius  and  1 -sigma  standard 
deviation  for  each  sphere.  The  localized  uncertainty  from  sphere  measurements  is  the 
sum  of  the  radial  errors  and  2-sigma  standard  deviation  for  these  measurements  (see 
Table  1).  Make  a whisker  plot  of  the  residual  errors  for  the  sphere  fit.  Alternatively,  plot 
the  residuals  of  the  sphere  fit  as  a function  of  angle  of  incidence  between  the  central  line- 
of-sight  for  the  system-under-test  and  the  normal  vector  on  the  sphere  (see  Appendix  E.3- 
B)  and  compute  the  regression  line  for  this  data. 

If  any  of  the  measured  points  fall  in  a forbidden  zone,  that  is,  physically  impossible 
regions  such  as  between  the  sphere  and  surface  of  the  ball  plate  due  to  measurement 
device  physics,  etc.,  these  points  should  also  be  reported. 


Table  1.  Best  Fit  Sphere 


Computed 

Radius 

Nominal 

Radius 

Standard 

Deviation 

(stdev) 

Uncertainty 
radial  error  + 

2 xstdev 

Sphere  1 

Sphere  2 

Sphere  3 

Sphere  4 

Sphere  5 

\ 

Ave.  Triplet 
Uncertainty 

wm% 

1 

Compute  the  point-to-point  distance  between  independent  balls  (i.e.,  use  each  ball  only 
once  in  the  computation)  on  opposite  sides  of  the  calibration  plate  (and  between  the  ends 
of  the  ball  bar  if  used)  using  the  center  values  of  the  sphere  fit.  The  scale  is  the  ratio  of 
the  longest  nominal  distance  to  the  current  distance.  This  is  used  to  normalize  the 
differences  to  the  longest  interball  distance.  Compute  the  standard  deviation  of  the  scaled 
difference  (see  Table  2).  Alternatively,  compute  the  least  square  fit  (6-degrees  of 
freedom)of  all  computed  sphere  center  locations  with  the  nominal  center  locations.  The 
standard  deviation  of  the  fit  is  the  standard  deviation  for  the  distortion  across  the  field. 
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Table  2 . Distance  Between  Spheres 


Distance  Between  Spheres 

Measured 

Nominal 

Difference 

Scale 

Normalized 

Difference 

Sphere  1 and  sphere  25 

Sphere  3 and  sphere  4 

Sphere  5 and  sphere  6 

• 

Ball  Bar 

Standard  Deviation 

4.0  RESULTS 

The  standard  deviation  of  the  sphere  fits  estimates  the  X-,  Y-,  Z-triplet  uncertainty,  uxyz. 
All  Max-Min  deviation  from  the  ideal  spheres  must  be  within  the  maximum  permissible 
triplet  error  (MPE),  if  one  is  provided.  In  the  event  that  the  specification  is  in  terms  of 
expanded  uncertainty  (2-sigma),  at  most,  5 % of  the  measured  deviations  to  the  sphere 
can  be  outside  the  expanded  uncertainty.  Note  any  significant  structure  to  the  sphere  fit 
such  as  oblateness  and  ellipticity.  If  the  regression  line  was  plotted,  then  the  slope  must 
be  less  than  0.05.  Error  may  be  due  to  asymmetry  in  the  structured  light  on  the  surface  of 
the  hemisphere,  as  seen  by  the  camera,  as  the  incidence  angle  gets  larger.  Error  may  also 
be  due  to  power  loss  as  the  incidence  angle  gets  larger.  This  test  may  identify  the 
presence  of  a potential  problem  but  does  not  diagnose  the  source. 

The  standard  deviation  for  the  normalized  distances  between  spheres  estimates  the 
distortion  in  the  volume  measurement,  upt-pt.  All  normalized  differences  must  be  within 
the  maximum  permissible  error  for  point  to-point  measurements,  if  one  is  supplied. 
Alternatively,  the  normalized  difference  must  be  less  than  the  single  point  uncertainty 
multiplied  by  fi  (this  assumes  the  pair  has  equivalent  and  spherical  uncertainties).  In 
the  event  that  the  specification  is  in  terms  of  expanded  uncertainty,  (2-sigma),  twice  the 
standard  deviation  for  the  normalized  difference  must  be  less  than  the  expanded 
uncertainty. 

The  uncertainty  of  the  calibration  plate  in  this  procedure  has  been  ignored.  Thermal 
expansion  of  the  balls  and  calibration  plate  are  generally  very  small  over  a wide  range  of 
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temperatures  (see  Appendix  E.3-C),  therefore  calibration  can  be  performed  in  a factory 
environment.  Thermal  errors,  when  they  occur,  will  most  likely  affect  the  mechanical 
stability  of  the  orientation  between  the  structured  light  generator  and  the  camera  system. 
The  system  performance  evaluation  should  identify  the  extended  validity  conditions  for 
temperature  of  use. 

5.0  CONCLUSION 

The  calibration  process  must  separately  identify  the  triplet  uncertainty  and  the  instrument 
uncertainty: 

Uxyz 

Upt-pt 

and  the  temperature  range  over  which  the  measurements  are  valid. 

If  the  structured  light  system  is  part  of  a fixed  installation,  then  the  instrument  uncertainty 
should  include  any  uncertainty  associated  with  translation  and  rotation  stages. 


5 


The  sphere  numbers  are  examples.  The  actual  choice  of  spheres  should  span  the  ball  plate. 
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Appendix  E.3-A  Calibration  Artifact 

The  artifact  shown  in  figure  A1  is  a ball  plate,  406.4  mm  (16  in.)  on  the  long  side  with  20 
hemispheres  held  in  three-point  mounts.  The  roundness  of  the  hemispheres  is  better  than 
0.00254  mm  (0.0001  in.).  The  locations  of  the  hemispheres  are  calibrated  less  than 
0.0127  mm  (0.0005  in.).  Some  methods  for  computing  ball  plate  calibration  artifact 
parameters  automatically  rely  on  random  placement  of  the  balls  so  correlation  can  be 
used  to  match  nominal  separations  with  actual  measurements. 


Figure  Ah  Diagram  of  Ball  Plate 
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Appendix  E.3-B  Sphere  Fit  and  Angle  of  Incidence  Calculation 

The  plotting  of  data  as  a function  of  angle  of  incidence  is  described  (see  figure  Bl). 

Compute  the  errors  for  each  measurement  to  a sphere,  using  constrained  (true)  radius:  5i 

Compute  the  center  of  the  sphere  based  on  all  measured  data:  C 

Compute  the  vector  for  the  origin  of  the  metrology  system  to  the  sphere  center:  OC. 

Compute  the  normal  vector  for  each  measured  point:  PC. 

Compute  the  incidence  angle,  0,  from  the  dot  product  of  the  vectors  OC  and  PC: 


0 = arccos 


Plot  the  sphere  error,  5i,  with  respect  to  the  incidence  angle  (see  figure  B2).  Ideally,  the  slope  of 
any  linear  regression  through  all  of  the  points  should  be  “flat”. 


Origin  of  Metrology  System,  O 


Computed  Sphere  Center,  C 


Measured  Sphere  Point,  P 


Figure  B1,  Diagram  of  Sphere  Measurement 
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Error  from  Sphere  (inches) 


Angle  of  Incidence  (degrees) 

0 10  20  30  40  50  60  70  80  90 


Figure  B2 . Typical  Plot  of  Incidence  Angle  Versus  Error . 
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Appendix  E.3-C  Calibration  Plate  Uncertainty 

The  compensation  / calibration  for  structured  light  measurement  systems  is  designed  to 
take  place  in  a shop  environment.  The  accuracy  of  this  calibration  is  contingent  on  the 
accuracy  of  the  standard  used  for  the  calibration.  The  calibration  plate  is  provided  with  a 
certification  at  a specific  temperature  and  must  be  adjusted  for  the  temperature  in  the 
shop  during  the  compensation  / calibration  process.  The  calibration  ball  plate  purchased 
from  Bal  Tec  is  fabricated  with  a cast  base,  meaning  that  the  thermal  expansion  is 
reasonably  isotropic,  therefore  the  linear  thermal  expansion  model  can  be  extended  to 
three  dimensions. 

Given  that  the  CTE  and  uncertainties  for  dimension  and  temperature  when  the  calibration 
ball  plate  is  used: 

CTEpiate  = 12  ppm  / °F 
Lpiate  - 406.4  mm  (16  in.) 

Upiate  = 0.00127  mm  (0.00005  in.) 

DiameterBalls  = 19.1  mm  (0.750  in.) 

CTEZro2  = 12  ppm  / °F 

Ueaiis  - 0.00127  mm  (0.00005  in.) 

UTemp  = 0.1°C  (0.2  °F) 


The  dimensional  correction  is  shown  in  figure  C 1 : 


Temperature  (deg  F) 

Plate 

Balls 


Figure  Cl.  Temperature  Correction  to  Dimensions 
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There  is  no  need  to  thermally  correct  for  ball  diameter  since  the  uncertainty  is  much  less 
than  the  uncertainty  of  the  structured  light  system  under  test.  For  a temperature  change 
of  2.2  °C  (4  °F)  there  is  also  no  need  to  correct  for  thermal  expansion  of  the  ball  plate 
(though  advisable  since  it  is  so  simple),  since  its  thermal  expansion  is  also  much  less  than 
uncertainty  of  the  structure  light  system  under  test. 

The  corresponding  dimensional  uncertainty  is  shown  in  figure  C2: 


Balls 

Figure  C2.  Calibration  Ball  Plate  Uncertainty 

The  uncertainties  are  substantially  less  than  the  uncertainty  of  the  measurement  system 
under  test  for  a ±10°  temperature  range,  roughly  a factor  of  10  less  than  the  measurement 
uncertainty.  Thus,  the  calibration  standard  will  maintain  acceptable  uncertainty  for  a 
wide  range  of  shop  temperatures. 
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APPENDIX  E.3-D  Uncertainty  of  Surface  Objects 

When  measuring  a single  point  location  several  times,  the  usual  procedure  is  to  compute 
the  average  point  location,  the  standard  deviation.  When  the  confidence  interval  for  the 
average  location  is  desired  the  standard  deviation  of  the  mean  is  computed.  The  standard 
deviation  of  the  mean  for  a group  of  points  on  a number  line  is  expressed  as: 


C 7 


where  a is  the  standard  deviation  for  the  group  of  points  and  n is  the  number  of  points  in 
the  group.  So,  given  cr=  0.010  and  n— 100,  the  standard  deviation  of  the  mean  (i.e.,  the 
confidence  interval  for  the  mean)  is  0.0254  mm  (0.001  in.).  It  is  also  easy  to  see  that 
halving  the  standard  deviation  for  the  group  of  points  is  equivalent  to  using  four  times  the 
number  of  points.  Thus,  as  the  number  of  measurements  increases,  the  confidence  in  the 
average  value  improves. 

This  concept  holds  true  for  any  average,  regardless  of  the  shape  of  the  distribution 
forming  the  average.  And  since  the  extraction  of  an  object  from  a cloud  of  points  is,  in 
effect,  an  average,  increasing  numbers  of  measurements  or  increasing  the  area  will 
improve  the  estimation  of  that  surface  object.  Clearly,  the  point  distribution  over  the 
surface  in  addition  to  the  uncertainty  of  the  individual  measurements  will  affect  the 
confidence  in  any  estimation.  This  appendix  derives  rules  of  thumb  for  the  "standard 
deviation  of  the  mean"  for  surface  objects  given  the  triplet  uncertainty,  number  of  points, 
and  area  of  measurements. 

The  technique  used  in  this  paper  is  a Monte  Carlo  (perturbation)  method.  A cloud  of 
points  representing  a typical  surface  object,  i.e.,  plane  or  hemisphere,  is  generated  using 
random  number  generators.  A random  uniform  distribution  is  used  over  the  surface  of 
the  object,  and  random  normal  distributions  are  used  in  and  out  of  the  object’s  surface. 
Then  that  cloud  of  points  is  fit  using  a least  squares  method  to  its  particular  object.  This 
process  is  repeated  hundreds  of  times  for  a particular  set  of  parameter,  e.g.,  surface  area, 
number  of  points,  and  measurement  uncertainty  for  the  cloud  of  points.  From  the 
hundreds  of  fittings  the  confidence  for  locating  the  surface  object  can  be  computed.  For 
example,  in  the  case  of  the  plane,  the  separation  of  two  planes  that  bound  68  % and  95  % 
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of  all  planes  computed  was  determined  (Appendix  E.3-A).  For  the  hemisphere,  the 
radius  of  the  sphere  containing  68  % and  95  % the  center  locations  was  computed 
(Appendix  E.3-B). 

Figures  1-3  show  the  results  for  fitting  point  clouds  to  planes.  As  might  be  anticipated, 
increasing  the  number  of  points  and  making  the  measurement  uncertainty  smaller 
improves  the  confidence  in  the  location  of  the  plane.  Increasing  the  area  of  the  plane 
does  not  increase  the  confidence  in  the  location  of  the  plane.  This  latter  observation 
seems  reasonable — the  larger  area  makes  the  orientation  of  the  plane  more  certain,  while 
the  increasing  radius  of  rotation  increases  the  displacement  of  the  plane,  thereby 
counteracting  each  other. 

The  plots  labeled  “theory”  were  empirically  derived — i.e.,  fitting  equations  to  the  68  % 
data.  Thus,  given  the  nominal  planar  bounds  for  one  plot  location,  the  planar  bounds  can 
be  estimated  at  all  other  points,  including  the  95  % data.  These  equations  are: 


From  these  equations,  the  effects  of  increasing  the  number  of  points  in  the  cloud  and 
improving  the  measurement  uncertainty  can  be  estimated.  For  example,  increasing  the 
number  of  measurements  in  a cloud  by  a factor  of  four  halves  the  uncertainty  of  the 
computed  plane.  This  is  similar  to  the  standard  deviation  of  the  mean  at  a point. 
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Nominal  Planar  Bounds  (tinches)  Nominal  Planar  Bounds  (iinches) 
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Figure  Dl.  ±0.01  inch  Measurement  Uncertainty,  400  inch2  Area 
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Figure  D2.  100  Points,  400  inch2  Area 
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Figure  D3.  100  Points,  ±0.01  inch  Measurement  Uncertainty 


It  should  be  noted  that  the  95  % bounds  are  not  twice  the  68  % bounds,  as  might  be 
expected  for  the  relationship  between  1 -sigma  and  2-sigma  values.  Figure  4 is  a typical 
histogram  for  a set  of  1000  planes  with  a particular  set  of  parameters.  A goodness-of-fit 
evaluation  proves  the  distribution  is  not  chi  squared  either,  too  much  data  are  shifted 
toward  cone  axis. 
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BOB  Chi  Squared 

Figure  D4.  Typical  Histogram  of  Bins  for  Bounding  Planes 

Figures  5-7  show  the  results  for  fitting  point  clouds  to  hemispheres.  The  region  of  points 
on  the  hemisphere  is  controlled  by  an  intersecting  cone  and  identified  by  the  cone  angle 
(i.e.,  the  angle  between  the  cone  axis  and  the  generatrix).  Again,  as  expected,  increasing 
the  number  of  points  and  making  the  measurement  uncertainty  smaller  improves  the 
confidence  in  locating  the  center  of  the  hemisphere.  This  time,  however,  increasing  the 
area  of  the  point  cloud  over  the  hemisphere  increases  the  confidence  in  the  location  of  the 
center  of  the  hemisphere. 
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Center  Uncertainty  (inches)  Center  Uncertainty  (inches) 


theory 


Figure  D5.  ±0.005  Measurement  Uncertainty,  40°  Cone  Angle 
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Figure  D6.  = 40°  Cone  Angle,  80  Points 
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Figure  D7.  ±0.005  Measurement  Uncertainty,  80  Points 


The  “theory”  values  were  generated  with  the  following  equations: 


[4] 

[5] 

[6] 


where  the  area  is  computed  from  the  zenith  angle:  2;r*(cos(0)-cos(^)).  Equations  [4]  and 
[5]  are  the  same  as  their  corresponding  equations  for  a plane,  [2]  and  [3].  Increasing  area 
decreases  the  uncertainty  of  the  hemisphere  center  point  as  shown  in  equation  [6].  It  is 
hypothesized  that  the  area  of  the  surface  becomes  a factor  as  the  curvature  is  increased. 
For  a plane,  there  is  no  curvature,  so  area  does  not  affect  the  outcome.  Or  stated  another 
way,  greater  curvature  causes  vectors  from  the  measured  locations  to  the  surface  to  have 
greater  variation,  thus  improving  the  fitting  process. 


For  the  hemisphere,  equations  [4]  through  [6]  can  be  combined: 
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[7] 


ct-,  n,-Area, 

<y  — = <j—  x — - 

c-  C1  cr1  y n2  ■ Area2 

This  equation  shows  the  interesting  result  that  given  an  arbitrarily  small  region  of  a 
sphere,  if  sufficient  numbers  of  points  are  measured,  the  uncertainty  of  the  center  can  also 
be  arbitrarily  small!  Whereas  the  usual  thought  is  that  measurement  of  small  regions  of  a 
sphere  cannot  accurately  identify  the  center  position. 

From  the  above,  it  appears  that  the  rules  of  thumb  are  largely  contained  in  equations  [2] 
and  [3]: 

• Increasing  the  number  of  points  decreases  the  mean  location  of  the  object  by  the 
square  root  of  the  ratio  of  points. 

• Changing  the  ratio  of  measurement  uncertainty  changes  the  mean  location  of  the 
object  proportionally. 

Obtaining  a rule  of  thumb  for  an  arbitrary  surface  area  may  not  be  feasible  due  to  the 
confounding  of  area  and  curvature.6  Likewise,  estimating  a nominal  cr  for  a 

surface  measured  by  a minimal  number  of  points  (i.e.,  n= 3 for  a plane,  n= 4 for  a sphere) 
and  unit  area  based  on  the  measurement  uncertainty  does  not  appear  feasible  because  of 
confounding  of  area  and  curvature. 


6 Need  a little  help  here  with  ideas. 
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APPENDIX  F:  CORPS  OF  ENGINEERS  FIELD  MEASUREMENT 

REQUIREMENTS 

by  Carlton  Daniel 
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Field  Measurement  Requirements 
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WMBBm 
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AMLIlft 


US  Army  Corps 
of  Engineers  ® 


Corps  Civil  Works  Program 


Provides  direct  public  service  benefits  for 

• Water  Control  Structures 
“ Pumping  Stations 
Navigational  Locks  & Dams 


WwW 

arm 


US  Army  Corps 
of  Engineers  ® 


Operations  and  Maintenance 
Challenges 


Almost  half  of  the  U.S.  inland  waterway  facilities 
are  nearing  or  extending  their  50-year  planned 
design  life 

Ensuring  expected  performance  levels  will  require 
rebuilding  or  replacing  many  existing  locks,  dams, 
hydropower  and  water  supply  facilities 

The  future  challenge  will  be  to  integrate  and  utilize 
laser  scanning  technologies  and  emerging 
calibration  standards  into  the  redesign  of  these 
structural  systems 
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US  Army  Corps 
of  Engineers  ® 


Multibeam  Soundings  & Laser  Image 
Data  Fusion 
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US  Army  Corps 
of  Engineers  ® 


Historical  Preservation 
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Mail 


US  Army  Corps 
of  Engineers  ® 


Structural  Assessment 


Length  26  Ft. 
Average  width  2.5  In. 
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APPENDIX  G:  METRICVISION  FACILITY  FOR  CALIBRATION  OF 
24M  FREQUENCY  MODULATED  COHERENT  LASER  RADAR 

by  Dave  Dozor 


NIST  IADAR  Calibration  Workshop 


MetricVision  Facility  for  Calibration  of 
24m  Frequency  Modulated  Coherent  laser  Radar 

June  12-13,  2003 
Gaithersburg,  MD 


• FM  Coherent  Laser  Radar 

• MV200  Product 


• Temperature  Controlled  Laboratory 

• Range  Calibration/Verification 

• Angle  Calibration/Verification 

• Future  Needs  for  Calibration 
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FM  Coherent  laser  Radar 


MetricVision  Laser  Radar  uses: 

• An  Infra-Red  (Xnom=1550nrn)  Laser  Diode 

• Frequency  Modulated  about  fnom  = cAnom 

• Modulation  is  linear 

• Local  Oscillators  of  controlled  length 

• Used  to  compare  with  Measurement  arm 

• Reference  Arm  of  controlled  length 

• Used  as  an  internal  "reference  length" 

• Detectors 


Laser  frequency  (wavelength)  is  linearly  modulated.  The  light  from  the  two 
paths  of  different  length  are  heterodyned.  The  resulting  difference 
frequency  is  proportional  to  the  OPLD  (and  therefore  range). 

Optical  Path  Length  Difference:  OPLD  = 2*L2  - 2*L1 
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Frequency 


200THZ 


laser  Tuning/Chirping 


transmitted  sigial 
received  signal 
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Serves  as  Internal 
Reference  Length. 

• Useful  for  ensuring 
laser  chirp  linearity  is 
maintained. 


6 

2491Hz  Signal 
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The  laser  Radar  System 


Basic  Concept  provides: 

1)  Single  transmit  and  return  path,  no 
shadowing 

2)  Non-contact  measurements  off  any  surface 

3)  Simultaneous  range  and  velocity 
measurements 

4)  Precise  measurements  (micron  precision) 

5)  Immunity  to  background  lighting 

2),  3)  Coherent  Detection 
4),  5)  Large  Tuning  BW 
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FM  laser  Radar  Product 


Prior  diagram  indicates; 

• Need  for  Temperature  Stability 

• Need  for  Beam  Focusing 

• Need  for  Beam  Pointing 

Other  affects  must  also  be  compensated, 

• Temperature,  Humidity,  and  Pressure  of  air. 

• Pointing  and  Focusing  imperfections. 

• PMA  and  AF  Kinematics  (Angle  Compensation) 


MV200  Product 


Final  Integration: 

• PMA 

• Laser  Radar 

• IOA/ Autofocus 

Must  compensate  for 
non-ideal  integration 

3-D  Prec.<2pprnrms 

5°C  < Top  < 40°C 

• Temperature  Comp'd 
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MV200  acquires  (R,  Az,  El)  data  to  <2ppmrms  precision 


• Range,  Azimuth,  Elevation  (spherical)  Data 
must  be  corrected  before  conversion  to 
X,Y,Z  (cartesian)  3-D  coordinates. 

• Range  comp'd  for: 

• Temp.,  Humidity  Press. 

• LO  length  settings 

• LOl,  L02  cross-over 

• Azimuth,  Elevation  comp'd  for: 

• Temperature 

• Machine  Kinematic  Parameters 
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Range  Compensation 


• Index  of  Refraction  (n)  of  Air  (Meas.  Arm) 


dT 

dn. 


= 1 ppm 


P,RH 


dP 


= 0.3  ppm 


T ,RH 


' dRH 


= 0.01  ppm 


T,P 


• Above  are  approximate  figures  (rules  of  thumb) 

• T in  °C,  P in  mBar,  RH  in  % 


• At  constant  temperature,  parameters: 

• Perpendicularity  of  axes 

• Coincidence  of  axes 

• Encoder  cycle  error 

• Many  others 

• These  parameters  are  determined  at  a set 
of  temperatures  and  "scheduled"  based  on 
RTD  feedback. 
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Temperature  Controlled  lab 


• MetricVision's  Temperature  Controlled  Lab 


• Volume: 

22,000  ft3  (623  m3) 

• Class: 

100,000 

• Temp  Rng: 

50°F  to  110°F  (10°C  to  43°C) 

• TempTol.: 

±0.5°F  (0.3°C) 

• Personnel: 

8 people  (max) 

• Heat  Load: 

50kWmax  (36HPmax) 

• Humidity: 

=±5%  (Estimate/Not  Controlled) 

• Pressure: 

=±2mbar  (Estimate/Not  Controlled) 
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Range  Calibration 


• Range  is  calibrated  against  an 
Environmentally  compensated: 

• Hewlett-Packard  5517C  Head  (He-Ne  Tube) 

• Hewlett-Packard  10780C  Receiver 

• Laser  Radar  and  Beam  expanded  HeNe 
are  retroreflected  off  of  a movable 
platform  and  compared  at  various  ranges. 


• The  range  calibration  parameters  are  used 
to  calibrate  the  photonic  circuit. 

* ^reference  arm  (4.0m  EOWA  typ.) 

* L|0cai_osciiiator_i  (8.5m  EOWA  typ.) 

* LocaLosciiiator_2  (15.5m  EOWA  typ.) 
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Range  Calibration  Photo 
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Range  Verification 


• The  Range  Calibration  must  be  verified  over  the 
operating  range. 

• Data  is  collected  on  LOl  and  L02  and  compared  to  the 
HeNe  Interferometer  Data. 

• 2 - 16m  at  0.5m  intervals 

• Figures  of  Merit  include: 

• STD  (RMS  error  to  HeNe  =>  All  sources  included) 

• PPM  (slope  wrt  HeNe  =^>  uncorrected  systematic  error) 

• Sep  (average  seperation  between  the  Los) 

• STD  must  be  less  than  12.5um  over  full  range 


Range  Verification  Results 


A Typical  FM  CLR 
• STD  = -5.69um 
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Angle  Compensation 


• Angle  Compensation  purpose: 

• Determine  mechanism  kinematic  parameters. 

• Utilizes  ranging  ability  of  the  FM  CLR  to 
measure  tooling  balls. 

• Based  on  interball  distance  being  constant: 

• Geometry  can  be  used  to  find  parameters 

• Various  views  are  recorded  and  model  applied 

• Results  show  corrected  interball  distances 
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Angle  Compensation 


• This  Extensive  Angle  Comp  is  performed 
at  various  temperature  to  schedule  the 
parameters. 

• Most  parameters  exhibit  no  temp  dependence 

• The  slope  is  determined  for  others 

• Our  embedded  software  compensates 


Angle  Comp/T-dependence 


• Top  plot  shows: 

• Perp  of  axes  (SN33) 

• 200nrad/°F 

• Very  good  fit 

60  65  70  75  80  85  90  95  100 

• Bottom  plot  shows: 

• Max  error  ~0.5prad 
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• After  compensating,  verification  is  done 

• Performed  over  a range  of  temperatures 

• Performed  on  5 to  8 views 

• Interball  distances  computed  to  assess 

• With  calibrated/compensated  CLR 

• End-on  ball  pipe  measurement  to  get  "truth" 

• Comparison  with  measured  interball  distances 

• Must  show  <50|jm  RMS  error  over  all  meas. 
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Angle  Comp  Verification 


• Top  plot  shows: 

• RMS  error  = 11pm 

• Max  error  = 31pm 

• Bottom  plot  shows: 

• Error  before  comp 

• View  centered  at: 

• (Az,EI)  = (45°, -69°) 

• Range  = 3m 

• One  View/One  Temp 
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• MetricVision  has  established  a Laser  Radar 
Calibration  Facility. 

• Used  for  Calibration  and  Verification  of  MV200 
Large-Volume  Coordinate  Metrology  Instrument 

• Temperature  Controlled  in  8 zones  to  0.5°F 

• Utilizes  a HeNe  Reference  and  Constant 
interball  distance  artifacts  for  calibration  of 
instrument  for  3D  measurements. 
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Range  Calibration  over  full  (Az,  El) 

• Precision  Electromechanics  to  position  scanner 
Very  Large,  Accurate  Truth  Model 

• ~15m  radius  artifact  with  appropriate  features 

• Scanner  can  measure  in/out  at  diff  locations 
Rail  Upgrades  (longer,  lateral  motion  trim) 
Non-vertical  Position  Calibration 

Larger  Temperature  Range  on  Facility 


• Range  Precision  vs.  Reflected  Energy 

• Continuous  Range  Monitoring 

• Environmental  & Seismic  Effects 
• Air  Turbulance 


• Temperature  Gradients 

• Vibration  Amplitude  and  Frequency  effects  on 
LADAR  (chirp  frequency,  doppler,  etc.) 
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APPENDIX  H:  CALIBRATION  VERIFICATION  FOR  LONG  RANGE 
TRIPOD-BASED  TERRESTRIAL  LASER  SCANNERS 


by  Eric  Martin 


Opl 

tech_| 

Calibration  Verification  for 

Long  Range  Tripod-based  Terrestrial 

Laser  Scanners 

Eric  Martin 

Product  Manager 

Optech,  Incorporated. 

Presented  by:  Eric  Martin 

Presented  to:  NIST  LADAR  Workshop 

Location:  Gaithersburg,  MD 

Date:  June  13,  2003 

Filename:  WAltmsl  \s'imaging'LI_Sales\Presentations\In_Progress\NIST'Drafls',Scanner_Cal_Verification.ppt 
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Abstract  / Intro 


• Terrestrial  laser  scanners,  like  all 
pieces  of  measuring  equipment  rely 
on  the  manufacturer’s  calibration 
process  to  ensure  the  product  meets 
specification  at  the  time  of  shipment. 

• While  the  actual  calibration 
procedures  may  vary  between 
manufacturer,  users  can  verify 
instrument  calibration  in  the  field, 
providing  they  understand  the  scanner 
requirements,  and  have  the  proper 
facilities  and  tools  available. 

© Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 
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Abstract/  Intro 

This  presentation  will  address  the  important  calibration 
attributes  as  they  pertain  to  long  range  terrestrial  laser 
scanners  (>  1 50m)  range,  and  outline  fhe  field 
procedures  that  can  be  employed  by  atypical  user  to 
verify  the  calibration  state  of  any  particular  seamier. 

This  procedure  is  not  meant  to  be  used  as  with  a type  or 
brand  of  scanner;  rather  it  can  be  adapted  to  serve  the 
needs  of  a wide  variety  of  products  currently  available. 

It  is  based  on  a practical  rather  than  theoretical 
approach. 


© Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 
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Tripod-based  Terrestrial  Laser  Scanners 


• General  attributes  - Long  range  (several  hundred  to 

> 1000m  range),  cm  to  mm  accuracy,  various  classes  of 
eyesafety. 

• Typically  send  a Light  pulse  (vis  or  IR)  through  a beam 
steering  system,  direct  it  to  a target,  collect  and  focus  the 
scattered  return  to  a receiver  / detector. 

• For  time-of-flight  systems,  a time  interval  is  calculated  and 
the  range  is  computed  from  the  known  speed  of  light  for 
the  wavelength  used  at  a given  atmospheric  condition. 

• No  matter  what  the  complexity  of  the  system,  most 
scanners  of  this  class  have  a single  common  element  - they 
produce  XYZ  point  cloud  data  from  diffuse  surfaces,  and 
include  a measure  of  intensity  of  the  signal  return. 

© Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 


Major  Markets  for  Tripod-based  Terrestrial  Scanners 

Used  primarily  to  scan  either: 

• Engineered  structures 

• Bridges,  Roadways,  Buildings, 

Dams,  Towers,  or  Process  Plants 

• Landscape  / Terrain 

• Open  pit  mines, 
golf  courses  greens  / faiiways, 
urban  planning, 

DTM  / DEM 
applications. 


© Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 
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Chief  Attributes 

I 

• Tripod  based  terrestrial  lidar  systems  are  not  designed 

for  a single  fixed  range 

i 

They  have  hundreds  or  even  thousands  of  meters  of 

ranging  capability 

m 

Their  versatility  is  extended  by  being  able  to  operate  at 

w 

relatively  short  range  (indoor  applications) 

\ 

\ 

© Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 

OptecKj  Chief  Attributes  - Differences  From 

Conventional  Instruments 

• Total  stations  are  typically  the  first  instruments  that 
come  to  mind  when  thinking  of  a method  to  verify 
scanner  calibration. 

Although  most  scanners  do  use  2 angles  and  a range  as 
their  source  polar  data,  they  do  not  output  this  in  a 
useable  format. 

• The  format  that  is  output,  used  and  manipulated  by  the 
customer  are  Cartesian  XYZ  point  clouds. 

• In  addition,  totalstations  do  not  record  the  intensity  of 
the  return  (as  intended  to  be  processed  as  part  of  a point 
cloud),  which  is  oftentimes  a very  valuable  element  of  a 
laser  seamier’ s output. 


) Copyright  2003.  Optecli  Incorporated.  All  rights  reserved. 
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Data  Output 

It  makes  the  most  sense  for  users  to  confine  their 
analysis  to  the  XYZi  or  XYZRGB  Cartesian  output  from 
a laser  seamier. 

It  is  the  fundamental  output  from  the  scanner,  It  is  the 
only  data  used  in  post  processing,  and  direct  cross- 
comparisons between  scanner  types  can  be  made 
regardless  of  the  make,  model  or  architecture. 

Cartesian  output  is  not  dependent  on  the  internal 
architecture  of  the  hardware;  future  scanners  deploying 
non-traditional  technologies  can  be  evaluated  and 
compared  directly  to  legacy  systems. 


) Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 
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Accuracy  Elements 

Accuracy  (as  defined  by  the  manufacturer)  is  the  single 
most  significant  element  of  calibration  requiring 
verification.  It  is  non-trivial  to  assess  accuracy  of  such 
scanning  system,  due  to  a large  number  of  factors  that 
are  not  present  in  conventional  instruments.  These  error 
sources  include 

Angular,  long  range, large  image  space,  variations  in 
material  reflectivity,  beam  incidence  to  the  target, 
etc... 


\ © Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 
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Op 

tech . Definition  of  Terms 

ii. 

• Rather  than  apply  generic  or  statistical  definitions  to 

these  terms,  the  ones  below  are  defined  to  best  address 

the  condition  that  applies  in  to  a laser  scanner  in  actual 

field  use  conditions. 

• They  are  based  on  analysis  of  the  full  amount  of  data 

produced  by  the  system,  using  the  available  analysis 

tools.  Bearing  this  in  mind,  the  following  definitions 

| apply  to  ILRIS-3D  in  typical  field  use. 

jt  ; - 

| © Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 

Target  Registration  Accuracy 

The  RMS  error  of  scan  comparing  scanner  derived 
target  locations  to  corresponding  survey  locations. 

The  total  error  is  the  sum  of  all  of  the  error  of  the 
instrument,  and  includes  all  error  groundtruth  as  well. 
This  is  expressed  as  a sphere  of  error  in  XYZ  space. 

This  is  a foundation  element  in  dealing  with  seamier 
accuracy,  as  it  typically  has  the  most  significant  effect 
when  scanning  over  large  field-of-view  and  large 
dynamic  ranges. 


) Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 
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Other  Terms  (cont’d) 


Optech  1 


• Modeling  Accuracy 

• The  error  in  3D  object  reconstruction  (real  vs 
reconstructed).  May  include  multiple  scan  alignment  errors, 
and  may  be  based  on  either  primitive  or  surface  (polygonal) 
modeling. 

• Resolution 


• The  minimum  distinguishable  feature  in  all  3 dimensions 
(or  in  a direction  specified  by  the  manufacturer). 


<D  Copyright  2003.  Optech  incorporated.  All  rights  reserved. 
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Other  Terms  (cont’d) 

Precision 

• Measure  of  consistency  in  output.  Just  as  a 
chronometer  can  keep  very  good  time,  and  is 
therefore  precise,  if  it  is  offset  from  the  atomic 
timeserver  by  some  bias  error,  it  is  not  accurate.  So 
too  can  the  output  from  a scanner  can  be  very  precise 
but  have  a bias  error  that  affects  its  accuracy 


) Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 
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Op  tech  | Typical  Indoor  Target  Wall  - Attributes 

and  Procedure 


Targets  used  must  conform  to  the  manufacturer’s 
recommendation 

Targets  are  very  accurately  surveyed 
by  traditional  means,  and  processed 
into  XYZ  Cartesian  groundtmth. 

User  mounts  scanner  at  a point  that 
allows  for  the  scan  to  cover  the 
instruments  entire  field  of  regard 


© Copyright  2003.  Optech  Incorporated.  All  rights  reserved. 
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Method  of  Verifying  Target  Registration 

Accuracy 

Indoor  Facility 

Many  indoor  facilities  already  exist  for  calibrating  2D 
cameras.  With  additional  modifications,  the  same  facilities 
can  be  used  as  a partial  verification  of  scanner  calibration 

They  can  be  made  to  address  other  issues  such  as  angle  of 
incidence  of  beam  to  target,  as  well  as  range  / intensity  errors 
due  to  material  reflectance  (color). 
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Data  Collection  and  Analysis 


Op  tech] 


• The  manufacturer’s  supplied  software  is  used  to  extract 
and  compute  the  centroid  of  the  target  locations  based 
either  on  intensity,  geometry,  or  both. 


1 


• Target  centroids  from 
scanner  (coordinate 
system  A)  are  then  compared 
with  survey  data  (coordinate 
system  B)  by  one  of  2 
methods 
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T Data  Collection  and  Analysis  (cont’d) 

1)  Resection 

• Resecting  coordinate  system  A-B  (or  B-A)  using  a least- 
squared  adjustment  on  all  targets 

2)  Vector  Comparison  . "T  v 

• Leaving  the 
datasets  in  their 
native  coordinate 
system,  and 
comparing  pairs 
of  points  in  one 
set  to 

corresponding  pairs 
points  in  the  other. 
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Typical  Outdoor  Facility 


Chosen  correctly,  Outdoor  facilities  have  the  advantage 
of  increased  range,  wider  dynamic  range  and  use  of 
oblique  surfaces,  and  provide  real-world  field  operating 
conditions. 


Data  collection  and  analvsis  is  typicallv  the  same  as 
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Calibrated  EDM  Baseline 

Calibrated  EDM  baselines  have  the  advantage  of  having 
very  accurate  (sub-mm)  slope  distances  over  greater 
ranges,  and  typically  serve  as  total  station  calibration 
checks. 

Typical  baselines  certified  by  government  are  in  the 
200-5 00m  range  employing  4 to  6 reference  pillars. 


AnfaflA  Ministry  of 

untario  Natural  Resources 
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EDM  Baseline  Utility 


An  EDM  baseline  is  not  a replacement  for  target 
registration  accuracy  verification,  but  has 
complimentary  advantages,  as  a calibrated  EDM 
baseline  can  be  used  to: 

1)  Identify  errors  over  a longer  range  than  can  be 
typically  found  elsewhere 

2)  Identify  errors  in  the  scanner  zero  position,  by 
setting  up  over  a known  point 
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EDM  Baseline  - Attributes 

Monuments  consist  of  12”  dia.  concrete  pillars  anchored 
6’  below  grade  spaced  over  several  hundred  meters 


Bronze  cap  with  5/8-1 1 UNC  screw 
thread  mounted  on  top  for  positive 
target  center. 
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EDM  Baseline  - Typical  Procedure 

Position  targets  on  center  over  monument 
Set  instrument  on  center  over  reference  pillar 
Scan  an  area  bounded  by  the  target 
Analyze  data  using  appropriate  software  tools. 
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EDM  Baseline  - Data  Analysis 


• Slope  distance  corrections  must  be  made  to  compensate  any 
height  deltas  between  the  target  end  and  the  reference 
scanner  if  they  are  present 

• Two  sets  of  comparisons  can  be  made 

• Slope  groundtruth  error  from  the  absolute  scanner 
position  to  target 

• Slope  groundtruth  error  from  relative  from  one  target  to 
the  next. 
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Absolute  Range  trior  Summary 


Nominal 
Range  (m) 

RTFF 

Polar 

RTFF' 

Delta 

X 

Cartesian 

Y Z 

.1 

Groundtruth  and  Errors 

Groundtruth  Slope  Error  from  Groundtruth  Slope 
Range  (mm) 

72 

7272  516602 

72  61332258 

0 057477416 

-4  97976 

72  46915 

2068031 

72  669481 

72.6708 

13 

154 

0 

•0  111843436 

0 111843436 

o 

0 

0 

o 

1533696 

226 

22669  64646 

226  5846214 

•72  6152214 

-155442 

225  9975 

7 03934 

226  64081 

226340? 

Relative  Range  Error  Summary 


153.9696 
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Conclusion 

Although  long-range  tripod  based  terrestrial  laser  scanners 
share  some  common  attributes,  the  actual  procedure  for 
calibration  varies  to  address  specific  elements  of  each 
system. 

• However,  all  systems  provide  a common  data  product  - 
Cartesian  XYZ  point  cloud  data.  Users  of  any  system 
should  be  able  to  readily  validate  the  scanner’s 
calibration  condition  by  a combination  of: 

1)  Understanding  the  specification  being  tested 
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2)  Consulting  with  the  manufacturer  to  ensure  that 
the  proper  tools  (target  types,  software,  etc)  are 
used  in  the  analysis. 

1 I 

I 

• The  benefits  of  doing  this  allow  for  a neutral 

comparison  of  any  scanner  type  in  its  class,  and  a direct 
comparison  of  the  state  of  calibration  of  each  device. 
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